--- base_model: aubmindlab/bert-base-arabertv02 datasets: - akhooli/arabic-triplets-1m-curated-sims-len language: - ar library_name: sentence-transformers pipeline_tag: sentence-similarity tags: - sentence-transformers - transformers.js - transformers - sentence-similarity - feature-extraction - dataset_size:75000 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss - mteb model-index: - name: Omartificial-Intelligence-Space/Arabert-matro-v4 results: - dataset: config: ar name: MTEB MIRACLRetrieval (ar) revision: main split: dev type: miracl/mmteb-miracl metrics: - type: ndcg_at_1 value: 52.659 - type: ndcg_at_3 value: 54.201 - type: ndcg_at_5 value: 57.262 - type: ndcg_at_10 value: 60.909 - type: ndcg_at_20 value: 63.096 - type: ndcg_at_100 value: 65.472 - type: ndcg_at_1000 value: 66.362 - type: map_at_1 value: 34.763 - type: map_at_3 value: 47.166000000000004 - type: map_at_5 value: 50.541000000000004 - type: map_at_10 value: 52.884 - type: map_at_20 value: 53.809 - type: map_at_100 value: 54.371 - type: map_at_1000 value: 54.425000000000004 - type: recall_at_1 value: 34.763 - type: recall_at_3 value: 54.891 - type: recall_at_5 value: 63.427 - type: recall_at_10 value: 72.94800000000001 - type: recall_at_20 value: 79.739 - type: recall_at_100 value: 89.629 - type: recall_at_1000 value: 95.381 - type: precision_at_1 value: 52.659 - type: precision_at_3 value: 31.191999999999997 - type: precision_at_5 value: 22.631 - type: precision_at_10 value: 13.511999999999999 - type: precision_at_20 value: 7.5600000000000005 - type: precision_at_100 value: 1.7489999999999999 - type: precision_at_1000 value: 0.187 - type: mrr_at_1 value: 52.65879999999999 - type: mrr_at_3 value: 61.3202 - type: mrr_at_5 value: 62.7256 - type: mrr_at_10 value: 63.5495 - type: mrr_at_20 value: 63.8468 - type: mrr_at_100 value: 64.0019 - type: mrr_at_1000 value: 64.0151 - type: nauc_ndcg_at_1_max value: 33.727000000000004 - type: nauc_ndcg_at_1_std value: -8.3951 - type: nauc_ndcg_at_1_diff1 value: 42.856899999999996 - type: nauc_ndcg_at_3_max value: 31.8924 - type: nauc_ndcg_at_3_std value: -10.5359 - type: nauc_ndcg_at_3_diff1 value: 37.061699999999995 - type: nauc_ndcg_at_5_max value: 33.4875 - type: nauc_ndcg_at_5_std value: -10.6765 - type: nauc_ndcg_at_5_diff1 value: 36.0359 - type: nauc_ndcg_at_10_max value: 34.6486 - type: nauc_ndcg_at_10_std value: -9.0577 - type: nauc_ndcg_at_10_diff1 value: 35.9743 - type: nauc_ndcg_at_20_max value: 35.4931 - type: nauc_ndcg_at_20_std value: -6.7009 - type: nauc_ndcg_at_20_diff1 value: 35.713699999999996 - type: nauc_ndcg_at_100_max value: 36.0012 - type: nauc_ndcg_at_100_std value: -4.2715000000000005 - type: nauc_ndcg_at_100_diff1 value: 35.4983 - type: nauc_ndcg_at_1000_max value: 36.0081 - type: nauc_ndcg_at_1000_std value: -4.3189 - type: nauc_ndcg_at_1000_diff1 value: 35.5317 - type: nauc_map_at_1_max value: 21.6751 - type: nauc_map_at_1_std value: -15.731700000000002 - type: nauc_map_at_1_diff1 value: 43.7988 - type: nauc_map_at_3_max value: 28.2868 - type: nauc_map_at_3_std value: -14.222299999999999 - type: nauc_map_at_3_diff1 value: 38.7019 - type: nauc_map_at_5_max value: 30.6599 - type: nauc_map_at_5_std value: -12.9248 - type: nauc_map_at_5_diff1 value: 37.2657 - type: nauc_map_at_10_max value: 31.8415 - type: nauc_map_at_10_std value: -11.6641 - type: nauc_map_at_10_diff1 value: 37.0156 - type: nauc_map_at_20_max value: 32.2409 - type: nauc_map_at_20_std value: -10.7783 - type: nauc_map_at_20_diff1 value: 36.9096 - type: nauc_map_at_100_max value: 32.406800000000004 - type: nauc_map_at_100_std value: -10.1756 - type: nauc_map_at_100_diff1 value: 36.7973 - type: nauc_map_at_1000_max value: 32.414300000000004 - type: nauc_map_at_1000_std value: -10.1575 - type: nauc_map_at_1000_diff1 value: 36.7973 - type: nauc_recall_at_1_max value: 21.6751 - type: nauc_recall_at_1_std value: -15.731700000000002 - type: nauc_recall_at_1_diff1 value: 43.7988 - type: nauc_recall_at_3_max value: 26.936799999999998 - type: nauc_recall_at_3_std value: -13.4627 - type: nauc_recall_at_3_diff1 value: 32.9438 - type: nauc_recall_at_5_max value: 31.0982 - type: nauc_recall_at_5_std value: -11.7438 - type: nauc_recall_at_5_diff1 value: 28.775000000000002 - type: nauc_recall_at_10_max value: 33.1301 - type: nauc_recall_at_10_std value: -8.1628 - type: nauc_recall_at_10_diff1 value: 26.9624 - type: nauc_recall_at_20_max value: 36.6475 - type: nauc_recall_at_20_std value: 1.6697 - type: nauc_recall_at_20_diff1 value: 25.0124 - type: nauc_recall_at_100_max value: 43.3763 - type: nauc_recall_at_100_std value: 27.1535 - type: nauc_recall_at_100_diff1 value: 19.642 - type: nauc_recall_at_1000_max value: 55.1331 - type: nauc_recall_at_1000_std value: 55.271300000000004 - type: nauc_recall_at_1000_diff1 value: 7.0743 - type: nauc_precision_at_1_max value: 33.727000000000004 - type: nauc_precision_at_1_std value: -8.3951 - type: nauc_precision_at_1_diff1 value: 42.856899999999996 - type: nauc_precision_at_3_max value: 30.201800000000002 - type: nauc_precision_at_3_std value: 4.3144 - type: nauc_precision_at_3_diff1 value: 11.6558 - type: nauc_precision_at_5_max value: 28.670299999999997 - type: nauc_precision_at_5_std value: 9.1624 - type: nauc_precision_at_5_diff1 value: 2.0577 - type: nauc_precision_at_10_max value: 25.1898 - type: nauc_precision_at_10_std value: 15.1474 - type: nauc_precision_at_10_diff1 value: -3.8282999999999996 - type: nauc_precision_at_20_max value: 22.163 - type: nauc_precision_at_20_std value: 20.707 - type: nauc_precision_at_20_diff1 value: -8.2827 - type: nauc_precision_at_100_max value: 16.6022 - type: nauc_precision_at_100_std value: 27.6889 - type: nauc_precision_at_100_diff1 value: -13.949 - type: nauc_precision_at_1000_max value: 13.256 - type: nauc_precision_at_1000_std value: 27.518900000000002 - type: nauc_precision_at_1000_diff1 value: -16.248099999999997 - type: nauc_mrr_at_1_max value: 33.727000000000004 - type: nauc_mrr_at_1_std value: -8.3951 - type: nauc_mrr_at_1_diff1 value: 42.856899999999996 - type: nauc_mrr_at_3_max value: 36.4097 - type: nauc_mrr_at_3_std value: -5.829199999999999 - type: nauc_mrr_at_3_diff1 value: 39.5664 - type: nauc_mrr_at_5_max value: 36.8858 - type: nauc_mrr_at_5_std value: -5.478899999999999 - type: nauc_mrr_at_5_diff1 value: 39.0891 - type: nauc_mrr_at_10_max value: 36.972300000000004 - type: nauc_mrr_at_10_std value: -5.2479 - type: nauc_mrr_at_10_diff1 value: 39.3498 - type: nauc_mrr_at_20_max value: 36.9505 - type: nauc_mrr_at_20_std value: -5.0094 - type: nauc_mrr_at_20_diff1 value: 39.342 - type: nauc_mrr_at_100_max value: 36.9136 - type: nauc_mrr_at_100_std value: -4.9965 - type: nauc_mrr_at_100_diff1 value: 39.3684 - type: nauc_mrr_at_1000_max value: 36.9076 - type: nauc_mrr_at_1000_std value: -5.0017 - type: nauc_mrr_at_1000_diff1 value: 39.3645 - type: main_score value: 60.909 task: type: Retrieval - dataset: config: ar name: MTEB MIRACLRetrievalHardNegatives (ar) revision: 95c8db7d4a6e9c1d8a60601afd63d553ae20a2eb split: dev type: mteb/miracl-hard-negatives metrics: - type: ndcg_at_1 value: 53.6 - type: ndcg_at_3 value: 55.553 - type: ndcg_at_5 value: 58.681000000000004 - type: ndcg_at_10 value: 62.61600000000001 - type: ndcg_at_20 value: 65.12 - type: ndcg_at_100 value: 67.56 - type: ndcg_at_1000 value: 68.426 - type: map_at_1 value: 35.994 - type: map_at_3 value: 48.586 - type: map_at_5 value: 51.919000000000004 - type: map_at_10 value: 54.459999999999994 - type: map_at_20 value: 55.531 - type: map_at_100 value: 56.155 - type: map_at_1000 value: 56.208999999999996 - type: recall_at_1 value: 35.994 - type: recall_at_3 value: 55.932 - type: recall_at_5 value: 64.78 - type: recall_at_10 value: 75.128 - type: recall_at_20 value: 82.818 - type: recall_at_100 value: 92.743 - type: recall_at_1000 value: 98.375 - type: precision_at_1 value: 53.6 - type: precision_at_3 value: 31.900000000000002 - type: precision_at_5 value: 23.119999999999997 - type: precision_at_10 value: 13.94 - type: precision_at_20 value: 7.920000000000001 - type: precision_at_100 value: 1.833 - type: precision_at_1000 value: 0.19499999999999998 - type: mrr_at_1 value: 53.6 - type: mrr_at_3 value: 62.5 - type: mrr_at_5 value: 64.105 - type: mrr_at_10 value: 64.9363 - type: mrr_at_20 value: 65.24799999999999 - type: mrr_at_100 value: 65.3865 - type: mrr_at_1000 value: 65.3988 - type: nauc_ndcg_at_1_max value: 32.5925 - type: nauc_ndcg_at_1_std value: -12.9576 - type: nauc_ndcg_at_1_diff1 value: 42.8949 - type: nauc_ndcg_at_3_max value: 28.2748 - type: nauc_ndcg_at_3_std value: -16.5509 - type: nauc_ndcg_at_3_diff1 value: 38.3185 - type: nauc_ndcg_at_5_max value: 29.7643 - type: nauc_ndcg_at_5_std value: -18.1028 - type: nauc_ndcg_at_5_diff1 value: 37.0747 - type: nauc_ndcg_at_10_max value: 31.4383 - type: nauc_ndcg_at_10_std value: -16.2253 - type: nauc_ndcg_at_10_diff1 value: 37.0354 - type: nauc_ndcg_at_20_max value: 31.6621 - type: nauc_ndcg_at_20_std value: -15.2722 - type: nauc_ndcg_at_20_diff1 value: 36.6883 - type: nauc_ndcg_at_100_max value: 32.5257 - type: nauc_ndcg_at_100_std value: -12.014800000000001 - type: nauc_ndcg_at_100_diff1 value: 36.8483 - type: nauc_ndcg_at_1000_max value: 32.2389 - type: nauc_ndcg_at_1000_std value: -12.2263 - type: nauc_ndcg_at_1000_diff1 value: 36.942 - type: nauc_map_at_1_max value: 19.910700000000002 - type: nauc_map_at_1_std value: -17.9674 - type: nauc_map_at_1_diff1 value: 44.4747 - type: nauc_map_at_3_max value: 24.4422 - type: nauc_map_at_3_std value: -20.0068 - type: nauc_map_at_3_diff1 value: 39.933 - type: nauc_map_at_5_max value: 26.8291 - type: nauc_map_at_5_std value: -19.9523 - type: nauc_map_at_5_diff1 value: 38.488099999999996 - type: nauc_map_at_10_max value: 28.4058 - type: nauc_map_at_10_std value: -18.604499999999998 - type: nauc_map_at_10_diff1 value: 38.2252 - type: nauc_map_at_20_max value: 28.610799999999998 - type: nauc_map_at_20_std value: -18.1148 - type: nauc_map_at_20_diff1 value: 38.0546 - type: nauc_map_at_100_max value: 28.8889 - type: nauc_map_at_100_std value: -17.4074 - type: nauc_map_at_100_diff1 value: 38.0604 - type: nauc_map_at_1000_max value: 28.886499999999998 - type: nauc_map_at_1000_std value: -17.3833 - type: nauc_map_at_1000_diff1 value: 38.0643 - type: nauc_recall_at_1_max value: 19.910700000000002 - type: nauc_recall_at_1_std value: -17.9674 - type: nauc_recall_at_1_diff1 value: 44.4747 - type: nauc_recall_at_3_max value: 22.2898 - type: nauc_recall_at_3_std value: -20.139599999999998 - type: nauc_recall_at_3_diff1 value: 34.8512 - type: nauc_recall_at_5_max value: 25.359399999999997 - type: nauc_recall_at_5_std value: -21.9072 - type: nauc_recall_at_5_diff1 value: 30.1171 - type: nauc_recall_at_10_max value: 29.2689 - type: nauc_recall_at_10_std value: -17.3282 - type: nauc_recall_at_10_diff1 value: 27.952700000000004 - type: nauc_recall_at_20_max value: 29.2543 - type: nauc_recall_at_20_std value: -15.187899999999999 - type: nauc_recall_at_20_diff1 value: 24.4768 - type: nauc_recall_at_100_max value: 38.3071 - type: nauc_recall_at_100_std value: 20.6617 - type: nauc_recall_at_100_diff1 value: 18.1309 - type: nauc_recall_at_1000_max value: 34.4722 - type: nauc_recall_at_1000_std value: 73.8318 - type: nauc_recall_at_1000_diff1 value: -19.3119 - type: nauc_precision_at_1_max value: 32.5925 - type: nauc_precision_at_1_std value: -12.9576 - type: nauc_precision_at_1_diff1 value: 42.8949 - type: nauc_precision_at_3_max value: 26.4374 - type: nauc_precision_at_3_std value: -2.0135 - type: nauc_precision_at_3_diff1 value: 10.7556 - type: nauc_precision_at_5_max value: 26.3901 - type: nauc_precision_at_5_std value: 1.3924 - type: nauc_precision_at_5_diff1 value: 0.8016000000000001 - type: nauc_precision_at_10_max value: 24.1182 - type: nauc_precision_at_10_std value: 10.337100000000001 - type: nauc_precision_at_10_diff1 value: -6.0842 - type: nauc_precision_at_20_max value: 19.0871 - type: nauc_precision_at_20_std value: 15.09 - type: nauc_precision_at_20_diff1 value: -11.8658 - type: nauc_precision_at_100_max value: 14.458499999999999 - type: nauc_precision_at_100_std value: 25.278699999999997 - type: nauc_precision_at_100_diff1 value: -16.281599999999997 - type: nauc_precision_at_1000_max value: 11.0229 - type: nauc_precision_at_1000_std value: 26.245 - type: nauc_precision_at_1000_diff1 value: -18.6688 - type: nauc_mrr_at_1_max value: 32.5925 - type: nauc_mrr_at_1_std value: -12.9576 - type: nauc_mrr_at_1_diff1 value: 42.8949 - type: nauc_mrr_at_3_max value: 34.6304 - type: nauc_mrr_at_3_std value: -10.3319 - type: nauc_mrr_at_3_diff1 value: 40.115 - type: nauc_mrr_at_5_max value: 34.7212 - type: nauc_mrr_at_5_std value: -10.3487 - type: nauc_mrr_at_5_diff1 value: 39.630900000000004 - type: nauc_mrr_at_10_max value: 35.0432 - type: nauc_mrr_at_10_std value: -10.0944 - type: nauc_mrr_at_10_diff1 value: 39.8161 - type: nauc_mrr_at_20_max value: 34.9258 - type: nauc_mrr_at_20_std value: -10.0789 - type: nauc_mrr_at_20_diff1 value: 39.928000000000004 - type: nauc_mrr_at_100_max value: 34.9594 - type: nauc_mrr_at_100_std value: -9.975000000000001 - type: nauc_mrr_at_100_diff1 value: 39.947700000000005 - type: nauc_mrr_at_1000_max value: 34.9435 - type: nauc_mrr_at_1000_std value: -9.9998 - type: nauc_mrr_at_1000_diff1 value: 39.9466 - type: main_score value: 62.61600000000001 task: type: Retrieval - dataset: config: ara-ara name: MTEB MLQARetrieval (ara-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 54.352000000000004 - type: ndcg_at_3 value: 63.162 - type: ndcg_at_5 value: 65.301 - type: ndcg_at_10 value: 67.56 - type: ndcg_at_20 value: 68.64 - type: ndcg_at_100 value: 70.456 - type: ndcg_at_1000 value: 71.073 - type: map_at_1 value: 54.352000000000004 - type: map_at_3 value: 60.992999999999995 - type: map_at_5 value: 62.173 - type: map_at_10 value: 63.11 - type: map_at_20 value: 63.408 - type: map_at_100 value: 63.664 - type: map_at_1000 value: 63.695 - type: recall_at_1 value: 54.352000000000004 - type: recall_at_3 value: 69.439 - type: recall_at_5 value: 74.66199999999999 - type: recall_at_10 value: 81.625 - type: recall_at_20 value: 85.88 - type: recall_at_100 value: 95.551 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 54.352000000000004 - type: precision_at_3 value: 23.146 - type: precision_at_5 value: 14.932 - type: precision_at_10 value: 8.161999999999999 - type: precision_at_20 value: 4.294 - type: precision_at_100 value: 0.9560000000000001 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 54.352000000000004 - type: mrr_at_3 value: 60.992900000000006 - type: mrr_at_5 value: 62.172799999999995 - type: mrr_at_10 value: 63.1095 - type: mrr_at_20 value: 63.4082 - type: mrr_at_100 value: 63.6643 - type: mrr_at_1000 value: 63.6952 - type: nauc_ndcg_at_1_max value: 63.8629 - type: nauc_ndcg_at_1_std value: 19.1249 - type: nauc_ndcg_at_1_diff1 value: 69.6701 - type: nauc_ndcg_at_3_max value: 65.7981 - type: nauc_ndcg_at_3_std value: 16.608999999999998 - type: nauc_ndcg_at_3_diff1 value: 64.7641 - type: nauc_ndcg_at_5_max value: 67.05600000000001 - type: nauc_ndcg_at_5_std value: 17.5816 - type: nauc_ndcg_at_5_diff1 value: 64.5078 - type: nauc_ndcg_at_10_max value: 66.51780000000001 - type: nauc_ndcg_at_10_std value: 15.823 - type: nauc_ndcg_at_10_diff1 value: 63.7149 - type: nauc_ndcg_at_20_max value: 66.586 - type: nauc_ndcg_at_20_std value: 16.6444 - type: nauc_ndcg_at_20_diff1 value: 63.906600000000005 - type: nauc_ndcg_at_100_max value: 66.08420000000001 - type: nauc_ndcg_at_100_std value: 17.3509 - type: nauc_ndcg_at_100_diff1 value: 64.8038 - type: nauc_ndcg_at_1000_max value: 66.05499999999999 - type: nauc_ndcg_at_1000_std value: 17.2498 - type: nauc_ndcg_at_1000_diff1 value: 65.0447 - type: nauc_map_at_1_max value: 63.8629 - type: nauc_map_at_1_std value: 19.1249 - type: nauc_map_at_1_diff1 value: 69.6701 - type: nauc_map_at_3_max value: 65.3096 - type: nauc_map_at_3_std value: 17.2977 - type: nauc_map_at_3_diff1 value: 65.93480000000001 - type: nauc_map_at_5_max value: 65.9495 - type: nauc_map_at_5_std value: 17.8282 - type: nauc_map_at_5_diff1 value: 65.8481 - type: nauc_map_at_10_max value: 65.7491 - type: nauc_map_at_10_std value: 17.1649 - type: nauc_map_at_10_diff1 value: 65.59469999999999 - type: nauc_map_at_20_max value: 65.7327 - type: nauc_map_at_20_std value: 17.3612 - type: nauc_map_at_20_diff1 value: 65.6271 - type: nauc_map_at_100_max value: 65.6799 - type: nauc_map_at_100_std value: 17.485500000000002 - type: nauc_map_at_100_diff1 value: 65.7529 - type: nauc_map_at_1000_max value: 65.6806 - type: nauc_map_at_1000_std value: 17.4828 - type: nauc_map_at_1000_diff1 value: 65.76610000000001 - type: nauc_recall_at_1_max value: 63.8629 - type: nauc_recall_at_1_std value: 19.1249 - type: nauc_recall_at_1_diff1 value: 69.6701 - type: nauc_recall_at_3_max value: 67.4174 - type: nauc_recall_at_3_std value: 14.2994 - type: nauc_recall_at_3_diff1 value: 60.91760000000001 - type: nauc_recall_at_5_max value: 71.4648 - type: nauc_recall_at_5_std value: 16.816200000000002 - type: nauc_recall_at_5_diff1 value: 59.445899999999995 - type: nauc_recall_at_10_max value: 70.0583 - type: nauc_recall_at_10_std value: 8.519599999999999 - type: nauc_recall_at_10_diff1 value: 54.1211 - type: nauc_recall_at_20_max value: 71.91969999999999 - type: nauc_recall_at_20_std value: 12.759200000000002 - type: nauc_recall_at_20_diff1 value: 53.408 - type: nauc_recall_at_100_max value: 67.7425 - type: nauc_recall_at_100_std value: 21.5154 - type: nauc_recall_at_100_diff1 value: 56.4753 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: 63.8629 - type: nauc_precision_at_1_std value: 19.1249 - type: nauc_precision_at_1_diff1 value: 69.6701 - type: nauc_precision_at_3_max value: 67.4174 - type: nauc_precision_at_3_std value: 14.2994 - type: nauc_precision_at_3_diff1 value: 60.91760000000001 - type: nauc_precision_at_5_max value: 71.4648 - type: nauc_precision_at_5_std value: 16.816200000000002 - type: nauc_precision_at_5_diff1 value: 59.445899999999995 - type: nauc_precision_at_10_max value: 70.0583 - type: nauc_precision_at_10_std value: 8.519599999999999 - type: nauc_precision_at_10_diff1 value: 54.1211 - type: nauc_precision_at_20_max value: 71.91969999999999 - type: nauc_precision_at_20_std value: 12.759200000000002 - type: nauc_precision_at_20_diff1 value: 53.408 - type: nauc_precision_at_100_max value: 67.7425 - type: nauc_precision_at_100_std value: 21.5154 - type: nauc_precision_at_100_diff1 value: 56.4753 - type: nauc_precision_at_1000_max value: 100.0 - type: nauc_precision_at_1000_std value: 100.0 - type: nauc_precision_at_1000_diff1 value: 100.0 - type: nauc_mrr_at_1_max value: 63.8629 - type: nauc_mrr_at_1_std value: 19.1249 - type: nauc_mrr_at_1_diff1 value: 69.6701 - type: nauc_mrr_at_3_max value: 65.3096 - type: nauc_mrr_at_3_std value: 17.2977 - type: nauc_mrr_at_3_diff1 value: 65.93480000000001 - type: nauc_mrr_at_5_max value: 65.9495 - type: nauc_mrr_at_5_std value: 17.8282 - type: nauc_mrr_at_5_diff1 value: 65.8481 - type: nauc_mrr_at_10_max value: 65.7491 - type: nauc_mrr_at_10_std value: 17.1649 - type: nauc_mrr_at_10_diff1 value: 65.59469999999999 - type: nauc_mrr_at_20_max value: 65.7327 - type: nauc_mrr_at_20_std value: 17.3612 - type: nauc_mrr_at_20_diff1 value: 65.6271 - type: nauc_mrr_at_100_max value: 65.6799 - type: nauc_mrr_at_100_std value: 17.485500000000002 - type: nauc_mrr_at_100_diff1 value: 65.7529 - type: nauc_mrr_at_1000_max value: 65.6806 - type: nauc_mrr_at_1000_std value: 17.4828 - type: nauc_mrr_at_1000_diff1 value: 65.76610000000001 - type: main_score value: 67.56 task: type: Retrieval - dataset: config: ara-deu name: MTEB MLQARetrieval (ara-deu) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 4.8309999999999995 - type: ndcg_at_3 value: 8.046000000000001 - type: ndcg_at_5 value: 9.417 - type: ndcg_at_10 value: 12.696 - type: ndcg_at_20 value: 14.901 - type: ndcg_at_100 value: 22.684 - type: ndcg_at_1000 value: 26.112000000000002 - type: map_at_1 value: 4.8309999999999995 - type: map_at_3 value: 7.166 - type: map_at_5 value: 7.915 - type: map_at_10 value: 9.267 - type: map_at_20 value: 9.873999999999999 - type: map_at_100 value: 10.883 - type: map_at_1000 value: 11.068 - type: recall_at_1 value: 4.8309999999999995 - type: recall_at_3 value: 10.628 - type: recall_at_5 value: 14.01 - type: recall_at_10 value: 24.154999999999998 - type: recall_at_20 value: 32.85 - type: recall_at_100 value: 75.845 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 4.8309999999999995 - type: precision_at_3 value: 3.543 - type: precision_at_5 value: 2.802 - type: precision_at_10 value: 2.415 - type: precision_at_20 value: 1.643 - type: precision_at_100 value: 0.758 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 4.8309 - type: mrr_at_3 value: 7.1659 - type: mrr_at_5 value: 7.9147 - type: mrr_at_10 value: 9.2667 - type: mrr_at_20 value: 9.8742 - type: mrr_at_100 value: 10.883 - type: mrr_at_1000 value: 11.068 - type: nauc_ndcg_at_1_max value: -25.577499999999997 - type: nauc_ndcg_at_1_std value: 39.2179 - type: nauc_ndcg_at_1_diff1 value: 37.3127 - type: nauc_ndcg_at_3_max value: -27.3836 - type: nauc_ndcg_at_3_std value: 26.8553 - type: nauc_ndcg_at_3_diff1 value: 12.25 - type: nauc_ndcg_at_5_max value: -27.642099999999996 - type: nauc_ndcg_at_5_std value: 21.423000000000002 - type: nauc_ndcg_at_5_diff1 value: 5.9791 - type: nauc_ndcg_at_10_max value: -28.704600000000003 - type: nauc_ndcg_at_10_std value: 22.2698 - type: nauc_ndcg_at_10_diff1 value: 9.443200000000001 - type: nauc_ndcg_at_20_max value: -26.114500000000003 - type: nauc_ndcg_at_20_std value: 20.3723 - type: nauc_ndcg_at_20_diff1 value: 9.768699999999999 - type: nauc_ndcg_at_100_max value: -25.0514 - type: nauc_ndcg_at_100_std value: 21.1662 - type: nauc_ndcg_at_100_diff1 value: 8.2169 - type: nauc_ndcg_at_1000_max value: -25.7957 - type: nauc_ndcg_at_1000_std value: 21.5751 - type: nauc_ndcg_at_1000_diff1 value: 11.3027 - type: nauc_map_at_1_max value: -25.577499999999997 - type: nauc_map_at_1_std value: 39.2179 - type: nauc_map_at_1_diff1 value: 37.3127 - type: nauc_map_at_3_max value: -27.502 - type: nauc_map_at_3_std value: 28.315099999999997 - type: nauc_map_at_3_diff1 value: 17.164299999999997 - type: nauc_map_at_5_max value: -27.709899999999998 - type: nauc_map_at_5_std value: 24.6866 - type: nauc_map_at_5_diff1 value: 12.922600000000001 - type: nauc_map_at_10_max value: -28.2524 - type: nauc_map_at_10_std value: 24.6336 - type: nauc_map_at_10_diff1 value: 14.2286 - type: nauc_map_at_20_max value: -27.3922 - type: nauc_map_at_20_std value: 23.7714 - type: nauc_map_at_20_diff1 value: 14.2366 - type: nauc_map_at_100_max value: -26.858500000000003 - type: nauc_map_at_100_std value: 23.663600000000002 - type: nauc_map_at_100_diff1 value: 14.1509 - type: nauc_map_at_1000_max value: -26.9102 - type: nauc_map_at_1000_std value: 23.7031 - type: nauc_map_at_1000_diff1 value: 14.3591 - type: nauc_recall_at_1_max value: -25.577499999999997 - type: nauc_recall_at_1_std value: 39.2179 - type: nauc_recall_at_1_diff1 value: 37.3127 - type: nauc_recall_at_3_max value: -27.0351 - type: nauc_recall_at_3_std value: 24.151600000000002 - type: nauc_recall_at_3_diff1 value: 2.3865999999999996 - type: nauc_recall_at_5_max value: -27.4151 - type: nauc_recall_at_5_std value: 15.4643 - type: nauc_recall_at_5_diff1 value: -6.840400000000001 - type: nauc_recall_at_10_max value: -29.447400000000002 - type: nauc_recall_at_10_std value: 19.253600000000002 - type: nauc_recall_at_10_diff1 value: 3.0009 - type: nauc_recall_at_20_max value: -23.4825 - type: nauc_recall_at_20_std value: 15.85 - type: nauc_recall_at_20_diff1 value: 4.4744 - type: nauc_recall_at_100_max value: -21.4006 - type: nauc_recall_at_100_std value: 19.050900000000002 - type: nauc_recall_at_100_diff1 value: -6.532399999999999 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: -25.577499999999997 - type: nauc_precision_at_1_std value: 39.2179 - type: nauc_precision_at_1_diff1 value: 37.3127 - type: nauc_precision_at_3_max value: -27.0351 - type: nauc_precision_at_3_std value: 24.151600000000002 - type: nauc_precision_at_3_diff1 value: 2.3865999999999996 - type: nauc_precision_at_5_max value: -27.4151 - type: nauc_precision_at_5_std value: 15.4643 - type: nauc_precision_at_5_diff1 value: -6.840400000000001 - type: nauc_precision_at_10_max value: -29.447400000000002 - type: nauc_precision_at_10_std value: 19.253600000000002 - type: nauc_precision_at_10_diff1 value: 3.0009 - type: nauc_precision_at_20_max value: -23.4825 - type: nauc_precision_at_20_std value: 15.85 - type: nauc_precision_at_20_diff1 value: 4.4744 - type: nauc_precision_at_100_max value: -21.4006 - type: nauc_precision_at_100_std value: 19.050900000000002 - type: nauc_precision_at_100_diff1 value: -6.532399999999999 - type: nauc_precision_at_1000_max value: 100.0 - type: nauc_precision_at_1000_std value: 100.0 - type: nauc_precision_at_1000_diff1 value: 100.0 - type: nauc_mrr_at_1_max value: -25.577499999999997 - type: nauc_mrr_at_1_std value: 39.2179 - type: nauc_mrr_at_1_diff1 value: 37.3127 - type: nauc_mrr_at_3_max value: -27.502 - type: nauc_mrr_at_3_std value: 28.315099999999997 - type: nauc_mrr_at_3_diff1 value: 17.164299999999997 - type: nauc_mrr_at_5_max value: -27.709899999999998 - type: nauc_mrr_at_5_std value: 24.6866 - type: nauc_mrr_at_5_diff1 value: 12.922600000000001 - type: nauc_mrr_at_10_max value: -28.2524 - type: nauc_mrr_at_10_std value: 24.6336 - type: nauc_mrr_at_10_diff1 value: 14.2286 - type: nauc_mrr_at_20_max value: -27.3922 - type: nauc_mrr_at_20_std value: 23.7714 - type: nauc_mrr_at_20_diff1 value: 14.2366 - type: nauc_mrr_at_100_max value: -26.858500000000003 - type: nauc_mrr_at_100_std value: 23.663600000000002 - type: nauc_mrr_at_100_diff1 value: 14.1509 - type: nauc_mrr_at_1000_max value: -26.9102 - type: nauc_mrr_at_1000_std value: 23.7031 - type: nauc_mrr_at_1000_diff1 value: 14.3591 - type: main_score value: 12.696 task: type: Retrieval - dataset: config: ara-eng name: MTEB MLQARetrieval (ara-eng) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 9.671000000000001 - type: ndcg_at_3 value: 14.005 - type: ndcg_at_5 value: 16.634999999999998 - type: ndcg_at_10 value: 19.897000000000002 - type: ndcg_at_20 value: 21.770999999999997 - type: ndcg_at_100 value: 26.640000000000004 - type: ndcg_at_1000 value: 30.848 - type: map_at_1 value: 9.671000000000001 - type: map_at_3 value: 12.959000000000001 - type: map_at_5 value: 14.42 - type: map_at_10 value: 15.770999999999999 - type: map_at_20 value: 16.267 - type: map_at_100 value: 16.908 - type: map_at_1000 value: 17.076 - type: recall_at_1 value: 9.671000000000001 - type: recall_at_3 value: 17.021 - type: recall_at_5 value: 23.404 - type: recall_at_10 value: 33.461999999999996 - type: recall_at_20 value: 41.006 - type: recall_at_100 value: 67.69800000000001 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 9.671000000000001 - type: precision_at_3 value: 5.6739999999999995 - type: precision_at_5 value: 4.681 - type: precision_at_10 value: 3.3459999999999996 - type: precision_at_20 value: 2.0500000000000003 - type: precision_at_100 value: 0.677 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 9.6712 - type: mrr_at_3 value: 12.959399999999999 - type: mrr_at_5 value: 14.419699999999999 - type: mrr_at_10 value: 15.7709 - type: mrr_at_20 value: 16.267 - type: mrr_at_100 value: 16.907700000000002 - type: mrr_at_1000 value: 17.075599999999998 - type: nauc_ndcg_at_1_max value: 34.8451 - type: nauc_ndcg_at_1_std value: 17.6999 - type: nauc_ndcg_at_1_diff1 value: 35.4275 - type: nauc_ndcg_at_3_max value: 32.306400000000004 - type: nauc_ndcg_at_3_std value: 12.6249 - type: nauc_ndcg_at_3_diff1 value: 23.2576 - type: nauc_ndcg_at_5_max value: 32.5065 - type: nauc_ndcg_at_5_std value: 14.9187 - type: nauc_ndcg_at_5_diff1 value: 21.5307 - type: nauc_ndcg_at_10_max value: 31.254199999999997 - type: nauc_ndcg_at_10_std value: 13.3432 - type: nauc_ndcg_at_10_diff1 value: 18.8432 - type: nauc_ndcg_at_20_max value: 31.003199999999996 - type: nauc_ndcg_at_20_std value: 14.116000000000001 - type: nauc_ndcg_at_20_diff1 value: 17.941399999999998 - type: nauc_ndcg_at_100_max value: 30.3153 - type: nauc_ndcg_at_100_std value: 14.8279 - type: nauc_ndcg_at_100_diff1 value: 17.1715 - type: nauc_ndcg_at_1000_max value: 31.6141 - type: nauc_ndcg_at_1000_std value: 14.8375 - type: nauc_ndcg_at_1000_diff1 value: 19.829900000000002 - type: nauc_map_at_1_max value: 34.8451 - type: nauc_map_at_1_std value: 17.6999 - type: nauc_map_at_1_diff1 value: 35.4275 - type: nauc_map_at_3_max value: 33.175399999999996 - type: nauc_map_at_3_std value: 13.6792 - type: nauc_map_at_3_diff1 value: 25.7374 - type: nauc_map_at_5_max value: 33.285599999999995 - type: nauc_map_at_5_std value: 15.162500000000001 - type: nauc_map_at_5_diff1 value: 24.4949 - type: nauc_map_at_10_max value: 32.588699999999996 - type: nauc_map_at_10_std value: 14.416699999999999 - type: nauc_map_at_10_diff1 value: 23.044999999999998 - type: nauc_map_at_20_max value: 32.4989 - type: nauc_map_at_20_std value: 14.6284 - type: nauc_map_at_20_diff1 value: 22.7093 - type: nauc_map_at_100_max value: 32.423 - type: nauc_map_at_100_std value: 14.796000000000001 - type: nauc_map_at_100_diff1 value: 22.5273 - type: nauc_map_at_1000_max value: 32.4971 - type: nauc_map_at_1000_std value: 14.8208 - type: nauc_map_at_1000_diff1 value: 22.686600000000002 - type: nauc_recall_at_1_max value: 34.8451 - type: nauc_recall_at_1_std value: 17.6999 - type: nauc_recall_at_1_diff1 value: 35.4275 - type: nauc_recall_at_3_max value: 30.145699999999998 - type: nauc_recall_at_3_std value: 10.1343 - type: nauc_recall_at_3_diff1 value: 17.414199999999997 - type: nauc_recall_at_5_max value: 30.787799999999997 - type: nauc_recall_at_5_std value: 14.5086 - type: nauc_recall_at_5_diff1 value: 15.271 - type: nauc_recall_at_10_max value: 28.5023 - type: nauc_recall_at_10_std value: 11.0234 - type: nauc_recall_at_10_diff1 value: 10.381400000000001 - type: nauc_recall_at_20_max value: 27.8892 - type: nauc_recall_at_20_std value: 13.3873 - type: nauc_recall_at_20_diff1 value: 8.2467 - type: nauc_recall_at_100_max value: 23.323 - type: nauc_recall_at_100_std value: 15.7918 - type: nauc_recall_at_100_diff1 value: 2.5512 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: 34.8451 - type: nauc_precision_at_1_std value: 17.6999 - type: nauc_precision_at_1_diff1 value: 35.4275 - type: nauc_precision_at_3_max value: 30.145699999999998 - type: nauc_precision_at_3_std value: 10.1343 - type: nauc_precision_at_3_diff1 value: 17.414199999999997 - type: nauc_precision_at_5_max value: 30.787799999999997 - type: nauc_precision_at_5_std value: 14.5086 - type: nauc_precision_at_5_diff1 value: 15.271 - type: nauc_precision_at_10_max value: 28.5023 - type: nauc_precision_at_10_std value: 11.0234 - type: nauc_precision_at_10_diff1 value: 10.381400000000001 - type: nauc_precision_at_20_max value: 27.8892 - type: nauc_precision_at_20_std value: 13.3873 - type: nauc_precision_at_20_diff1 value: 8.2467 - type: nauc_precision_at_100_max value: 23.323 - type: nauc_precision_at_100_std value: 15.7918 - type: nauc_precision_at_100_diff1 value: 2.5512 - type: nauc_precision_at_1000_max value: 100.0 - type: nauc_precision_at_1000_std value: 100.0 - type: nauc_precision_at_1000_diff1 value: 100.0 - type: nauc_mrr_at_1_max value: 34.8451 - type: nauc_mrr_at_1_std value: 17.6999 - type: nauc_mrr_at_1_diff1 value: 35.4275 - type: nauc_mrr_at_3_max value: 33.175399999999996 - type: nauc_mrr_at_3_std value: 13.6792 - type: nauc_mrr_at_3_diff1 value: 25.7374 - type: nauc_mrr_at_5_max value: 33.285599999999995 - type: nauc_mrr_at_5_std value: 15.162500000000001 - type: nauc_mrr_at_5_diff1 value: 24.4949 - type: nauc_mrr_at_10_max value: 32.588699999999996 - type: nauc_mrr_at_10_std value: 14.416699999999999 - type: nauc_mrr_at_10_diff1 value: 23.044999999999998 - type: nauc_mrr_at_20_max value: 32.4989 - type: nauc_mrr_at_20_std value: 14.6284 - type: nauc_mrr_at_20_diff1 value: 22.7093 - type: nauc_mrr_at_100_max value: 32.423 - type: nauc_mrr_at_100_std value: 14.796000000000001 - type: nauc_mrr_at_100_diff1 value: 22.5273 - type: nauc_mrr_at_1000_max value: 32.4971 - type: nauc_mrr_at_1000_std value: 14.8208 - type: nauc_mrr_at_1000_diff1 value: 22.686600000000002 - type: main_score value: 19.897000000000002 task: type: Retrieval - dataset: config: ara-spa name: MTEB MLQARetrieval (ara-spa) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 5.59 - type: ndcg_at_3 value: 7.306 - type: ndcg_at_5 value: 9.364 - type: ndcg_at_10 value: 11.623999999999999 - type: ndcg_at_20 value: 14.091000000000001 - type: ndcg_at_100 value: 23.613999999999997 - type: ndcg_at_1000 value: 25.846999999999998 - type: map_at_1 value: 5.59 - type: map_at_3 value: 6.832000000000001 - type: map_at_5 value: 7.981000000000001 - type: map_at_10 value: 8.945 - type: map_at_20 value: 9.597999999999999 - type: map_at_100 value: 10.728 - type: map_at_1000 value: 10.854 - type: recall_at_1 value: 5.59 - type: recall_at_3 value: 8.696 - type: recall_at_5 value: 13.665 - type: recall_at_10 value: 20.497 - type: recall_at_20 value: 30.435000000000002 - type: recall_at_100 value: 84.47200000000001 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 5.59 - type: precision_at_3 value: 2.899 - type: precision_at_5 value: 2.733 - type: precision_at_10 value: 2.0500000000000003 - type: precision_at_20 value: 1.522 - type: precision_at_100 value: 0.845 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 5.5901 - type: mrr_at_3 value: 6.832299999999999 - type: mrr_at_5 value: 7.9814 - type: mrr_at_10 value: 8.9448 - type: mrr_at_20 value: 9.5983 - type: mrr_at_100 value: 10.7277 - type: mrr_at_1000 value: 10.8543 - type: nauc_ndcg_at_1_max value: 30.457600000000003 - type: nauc_ndcg_at_1_std value: 30.5912 - type: nauc_ndcg_at_1_diff1 value: 15.1628 - type: nauc_ndcg_at_3_max value: 20.9981 - type: nauc_ndcg_at_3_std value: 34.5208 - type: nauc_ndcg_at_3_diff1 value: 9.3722 - type: nauc_ndcg_at_5_max value: 17.321 - type: nauc_ndcg_at_5_std value: 35.8298 - type: nauc_ndcg_at_5_diff1 value: 2.1479000000000004 - type: nauc_ndcg_at_10_max value: 12.1236 - type: nauc_ndcg_at_10_std value: 34.7457 - type: nauc_ndcg_at_10_diff1 value: 0.1303 - type: nauc_ndcg_at_20_max value: 9.1906 - type: nauc_ndcg_at_20_std value: 33.1524 - type: nauc_ndcg_at_20_diff1 value: -0.4109 - type: nauc_ndcg_at_100_max value: 11.9124 - type: nauc_ndcg_at_100_std value: 35.9542 - type: nauc_ndcg_at_100_diff1 value: 1.1774 - type: nauc_ndcg_at_1000_max value: 13.844000000000001 - type: nauc_ndcg_at_1000_std value: 35.4475 - type: nauc_ndcg_at_1000_diff1 value: 1.6099999999999999 - type: nauc_map_at_1_max value: 30.457600000000003 - type: nauc_map_at_1_std value: 30.5912 - type: nauc_map_at_1_diff1 value: 15.1628 - type: nauc_map_at_3_max value: 23.3595 - type: nauc_map_at_3_std value: 34.1498 - type: nauc_map_at_3_diff1 value: 10.738 - type: nauc_map_at_5_max value: 20.7141 - type: nauc_map_at_5_std value: 34.807500000000005 - type: nauc_map_at_5_diff1 value: 5.8309 - type: nauc_map_at_10_max value: 17.7238 - type: nauc_map_at_10_std value: 34.4915 - type: nauc_map_at_10_diff1 value: 4.4532 - type: nauc_map_at_20_max value: 16.7467 - type: nauc_map_at_20_std value: 34.1792 - type: nauc_map_at_20_diff1 value: 4.2383999999999995 - type: nauc_map_at_100_max value: 17.158 - type: nauc_map_at_100_std value: 34.8559 - type: nauc_map_at_100_diff1 value: 4.2197 - type: nauc_map_at_1000_max value: 17.3136 - type: nauc_map_at_1000_std value: 34.8147 - type: nauc_map_at_1000_diff1 value: 4.2817 - type: nauc_recall_at_1_max value: 30.457600000000003 - type: nauc_recall_at_1_std value: 30.5912 - type: nauc_recall_at_1_diff1 value: 15.1628 - type: nauc_recall_at_3_max value: 15.4831 - type: nauc_recall_at_3_std value: 35.2306 - type: nauc_recall_at_3_diff1 value: 6.203 - type: nauc_recall_at_5_max value: 10.8514 - type: nauc_recall_at_5_std value: 37.8489 - type: nauc_recall_at_5_diff1 value: -4.9883 - type: nauc_recall_at_10_max value: 2.8804 - type: nauc_recall_at_10_std value: 34.8463 - type: nauc_recall_at_10_diff1 value: -6.776699999999999 - type: nauc_recall_at_20_max value: -2.8364000000000003 - type: nauc_recall_at_20_std value: 30.512299999999996 - type: nauc_recall_at_20_diff1 value: -7.1777 - type: nauc_recall_at_100_max value: -0.6318 - type: nauc_recall_at_100_std value: 39.151 - type: nauc_recall_at_100_diff1 value: -1.2727 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: 30.457600000000003 - type: nauc_precision_at_1_std value: 30.5912 - type: nauc_precision_at_1_diff1 value: 15.1628 - type: nauc_precision_at_3_max value: 15.4831 - type: nauc_precision_at_3_std value: 35.2306 - type: nauc_precision_at_3_diff1 value: 6.203 - type: nauc_precision_at_5_max value: 10.8514 - type: nauc_precision_at_5_std value: 37.8489 - type: nauc_precision_at_5_diff1 value: -4.9883 - type: nauc_precision_at_10_max value: 2.8804 - type: nauc_precision_at_10_std value: 34.8463 - type: nauc_precision_at_10_diff1 value: -6.776699999999999 - type: nauc_precision_at_20_max value: -2.8364000000000003 - type: nauc_precision_at_20_std value: 30.512299999999996 - type: nauc_precision_at_20_diff1 value: -7.1777 - type: nauc_precision_at_100_max value: -0.6318 - type: nauc_precision_at_100_std value: 39.151 - type: nauc_precision_at_100_diff1 value: -1.2727 - type: nauc_precision_at_1000_max value: 100.0 - type: nauc_precision_at_1000_std value: 100.0 - type: nauc_precision_at_1000_diff1 value: 100.0 - type: nauc_mrr_at_1_max value: 30.457600000000003 - type: nauc_mrr_at_1_std value: 30.5912 - type: nauc_mrr_at_1_diff1 value: 15.1628 - type: nauc_mrr_at_3_max value: 23.3595 - type: nauc_mrr_at_3_std value: 34.1498 - type: nauc_mrr_at_3_diff1 value: 10.738 - type: nauc_mrr_at_5_max value: 20.7141 - type: nauc_mrr_at_5_std value: 34.807500000000005 - type: nauc_mrr_at_5_diff1 value: 5.8309 - type: nauc_mrr_at_10_max value: 17.7238 - type: nauc_mrr_at_10_std value: 34.4915 - type: nauc_mrr_at_10_diff1 value: 4.4532 - type: nauc_mrr_at_20_max value: 16.7467 - type: nauc_mrr_at_20_std value: 34.1792 - type: nauc_mrr_at_20_diff1 value: 4.2383999999999995 - type: nauc_mrr_at_100_max value: 17.158 - type: nauc_mrr_at_100_std value: 34.8559 - type: nauc_mrr_at_100_diff1 value: 4.2197 - type: nauc_mrr_at_1000_max value: 17.3136 - type: nauc_mrr_at_1000_std value: 34.8147 - type: nauc_mrr_at_1000_diff1 value: 4.2817 - type: main_score value: 11.623999999999999 task: type: Retrieval - dataset: config: ara-hin name: MTEB MLQARetrieval (ara-hin) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 1.075 - type: ndcg_at_3 value: 3.1109999999999998 - type: ndcg_at_5 value: 4.198 - type: ndcg_at_10 value: 4.886 - type: ndcg_at_20 value: 6.627 - type: ndcg_at_100 value: 15.818999999999999 - type: ndcg_at_1000 value: 20.21 - type: map_at_1 value: 1.075 - type: map_at_3 value: 2.688 - type: map_at_5 value: 3.2800000000000002 - type: map_at_10 value: 3.5589999999999997 - type: map_at_20 value: 4.024 - type: map_at_100 value: 5.065 - type: map_at_1000 value: 5.311 - type: recall_at_1 value: 1.075 - type: recall_at_3 value: 4.301 - type: recall_at_5 value: 6.988999999999999 - type: recall_at_10 value: 9.139999999999999 - type: recall_at_20 value: 16.128999999999998 - type: recall_at_100 value: 69.355 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 1.075 - type: precision_at_3 value: 1.434 - type: precision_at_5 value: 1.398 - type: precision_at_10 value: 0.914 - type: precision_at_20 value: 0.8059999999999999 - type: precision_at_100 value: 0.694 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 1.0753000000000001 - type: mrr_at_3 value: 2.6882 - type: mrr_at_5 value: 3.2796 - type: mrr_at_10 value: 3.5595000000000003 - type: mrr_at_20 value: 4.0238000000000005 - type: mrr_at_100 value: 5.0652 - type: mrr_at_1000 value: 5.3114 - type: nauc_ndcg_at_1_max value: 100.0 - type: nauc_ndcg_at_1_std value: 100.0 - type: nauc_ndcg_at_1_diff1 value: 24.8943 - type: nauc_ndcg_at_3_max value: 80.9159 - type: nauc_ndcg_at_3_std value: 59.406800000000004 - type: nauc_ndcg_at_3_diff1 value: 24.9941 - type: nauc_ndcg_at_5_max value: 60.6127 - type: nauc_ndcg_at_5_std value: 38.3677 - type: nauc_ndcg_at_5_diff1 value: 21.407300000000003 - type: nauc_ndcg_at_10_max value: 55.8719 - type: nauc_ndcg_at_10_std value: 30.0006 - type: nauc_ndcg_at_10_diff1 value: 16.256 - type: nauc_ndcg_at_20_max value: 40.9477 - type: nauc_ndcg_at_20_std value: 18.8914 - type: nauc_ndcg_at_20_diff1 value: 13.459999999999999 - type: nauc_ndcg_at_100_max value: 33.3277 - type: nauc_ndcg_at_100_std value: 21.4906 - type: nauc_ndcg_at_100_diff1 value: 7.316000000000001 - type: nauc_ndcg_at_1000_max value: 46.3494 - type: nauc_ndcg_at_1000_std value: 27.355 - type: nauc_ndcg_at_1000_diff1 value: 12.9588 - type: nauc_map_at_1_max value: 100.0 - type: nauc_map_at_1_std value: 100.0 - type: nauc_map_at_1_diff1 value: 24.8943 - type: nauc_map_at_3_max value: 82.5 - type: nauc_map_at_3_std value: 62.7763 - type: nauc_map_at_3_diff1 value: 24.9858 - type: nauc_map_at_5_max value: 68.398 - type: nauc_map_at_5_std value: 47.3023 - type: nauc_map_at_5_diff1 value: 22.983999999999998 - type: nauc_map_at_10_max value: 65.6939 - type: nauc_map_at_10_std value: 41.7441 - type: nauc_map_at_10_diff1 value: 20.024900000000002 - type: nauc_map_at_20_max value: 58.703399999999995 - type: nauc_map_at_20_std value: 35.993700000000004 - type: nauc_map_at_20_diff1 value: 18.6205 - type: nauc_map_at_100_max value: 55.6135 - type: nauc_map_at_100_std value: 35.5159 - type: nauc_map_at_100_diff1 value: 16.424500000000002 - type: nauc_map_at_1000_max value: 57.1497 - type: nauc_map_at_1000_std value: 36.3072 - type: nauc_map_at_1000_diff1 value: 17.0605 - type: nauc_recall_at_1_max value: 100.0 - type: nauc_recall_at_1_std value: 100.0 - type: nauc_recall_at_1_diff1 value: 24.8943 - type: nauc_recall_at_3_max value: 78.1251 - type: nauc_recall_at_3_std value: 53.4704 - type: nauc_recall_at_3_diff1 value: 25.0087 - type: nauc_recall_at_5_max value: 48.4202 - type: nauc_recall_at_5_std value: 24.9277 - type: nauc_recall_at_5_diff1 value: 18.715200000000003 - type: nauc_recall_at_10_max value: 42.2174 - type: nauc_recall_at_10_std value: 14.477699999999999 - type: nauc_recall_at_10_diff1 value: 10.6624 - type: nauc_recall_at_20_max value: 20.7685 - type: nauc_recall_at_20_std value: 0.758 - type: nauc_recall_at_20_diff1 value: 7.546 - type: nauc_recall_at_100_max value: 6.9487 - type: nauc_recall_at_100_std value: 9.4323 - type: nauc_recall_at_100_diff1 value: -4.2262 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: 100.0 - type: nauc_precision_at_1_std value: 100.0 - type: nauc_precision_at_1_diff1 value: 24.8943 - type: nauc_precision_at_3_max value: 78.1251 - type: nauc_precision_at_3_std value: 53.4704 - type: nauc_precision_at_3_diff1 value: 25.0087 - type: nauc_precision_at_5_max value: 48.4202 - type: nauc_precision_at_5_std value: 24.9277 - type: nauc_precision_at_5_diff1 value: 18.715200000000003 - type: nauc_precision_at_10_max value: 42.2174 - type: nauc_precision_at_10_std value: 14.477699999999999 - type: nauc_precision_at_10_diff1 value: 10.6624 - type: nauc_precision_at_20_max value: 20.7685 - type: nauc_precision_at_20_std value: 0.758 - type: nauc_precision_at_20_diff1 value: 7.546 - type: nauc_precision_at_100_max value: 6.9487 - type: nauc_precision_at_100_std value: 9.4323 - type: nauc_precision_at_100_diff1 value: -4.2262 - type: nauc_precision_at_1000_max value: .nan - type: nauc_precision_at_1000_std value: .nan - type: nauc_precision_at_1000_diff1 value: .nan - type: nauc_mrr_at_1_max value: 100.0 - type: nauc_mrr_at_1_std value: 100.0 - type: nauc_mrr_at_1_diff1 value: 24.8943 - type: nauc_mrr_at_3_max value: 82.5 - type: nauc_mrr_at_3_std value: 62.7763 - type: nauc_mrr_at_3_diff1 value: 24.9858 - type: nauc_mrr_at_5_max value: 68.398 - type: nauc_mrr_at_5_std value: 47.3023 - type: nauc_mrr_at_5_diff1 value: 22.983999999999998 - type: nauc_mrr_at_10_max value: 65.6939 - type: nauc_mrr_at_10_std value: 41.7441 - type: nauc_mrr_at_10_diff1 value: 20.024900000000002 - type: nauc_mrr_at_20_max value: 58.703399999999995 - type: nauc_mrr_at_20_std value: 35.993700000000004 - type: nauc_mrr_at_20_diff1 value: 18.6205 - type: nauc_mrr_at_100_max value: 55.6135 - type: nauc_mrr_at_100_std value: 35.5159 - type: nauc_mrr_at_100_diff1 value: 16.424500000000002 - type: nauc_mrr_at_1000_max value: 57.1497 - type: nauc_mrr_at_1000_std value: 36.3072 - type: nauc_mrr_at_1000_diff1 value: 17.0605 - type: main_score value: 4.886 task: type: Retrieval - dataset: config: ara-vie name: MTEB MLQARetrieval (ara-vie) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 11.043 - type: ndcg_at_3 value: 13.205 - type: ndcg_at_5 value: 15.421000000000001 - type: ndcg_at_10 value: 16.588 - type: ndcg_at_20 value: 19.336000000000002 - type: ndcg_at_100 value: 26.648 - type: ndcg_at_1000 value: 30.168 - type: map_at_1 value: 11.043 - type: map_at_3 value: 12.679000000000002 - type: map_at_5 value: 13.875000000000002 - type: map_at_10 value: 14.344000000000001 - type: map_at_20 value: 15.076 - type: map_at_100 value: 16.021 - type: map_at_1000 value: 16.219 - type: recall_at_1 value: 11.043 - type: recall_at_3 value: 14.724 - type: recall_at_5 value: 20.244999999999997 - type: recall_at_10 value: 23.926 - type: recall_at_20 value: 34.969 - type: recall_at_100 value: 75.46000000000001 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 11.043 - type: precision_at_3 value: 4.9079999999999995 - type: precision_at_5 value: 4.0489999999999995 - type: precision_at_10 value: 2.393 - type: precision_at_20 value: 1.748 - type: precision_at_100 value: 0.755 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 11.0429 - type: mrr_at_3 value: 12.6789 - type: mrr_at_5 value: 13.8753 - type: mrr_at_10 value: 14.3437 - type: mrr_at_20 value: 15.0759 - type: mrr_at_100 value: 16.020899999999997 - type: mrr_at_1000 value: 16.2188 - type: nauc_ndcg_at_1_max value: 43.7806 - type: nauc_ndcg_at_1_std value: 36.2659 - type: nauc_ndcg_at_1_diff1 value: 15.229999999999999 - type: nauc_ndcg_at_3_max value: 47.7917 - type: nauc_ndcg_at_3_std value: 41.8048 - type: nauc_ndcg_at_3_diff1 value: 12.288 - type: nauc_ndcg_at_5_max value: 43.2288 - type: nauc_ndcg_at_5_std value: 37.7581 - type: nauc_ndcg_at_5_diff1 value: 8.984200000000001 - type: nauc_ndcg_at_10_max value: 45.9003 - type: nauc_ndcg_at_10_std value: 41.0103 - type: nauc_ndcg_at_10_diff1 value: 8.162700000000001 - type: nauc_ndcg_at_20_max value: 44.3326 - type: nauc_ndcg_at_20_std value: 40.3511 - type: nauc_ndcg_at_20_diff1 value: 5.3708 - type: nauc_ndcg_at_100_max value: 46.945100000000004 - type: nauc_ndcg_at_100_std value: 43.0094 - type: nauc_ndcg_at_100_diff1 value: 2.3986 - type: nauc_ndcg_at_1000_max value: 45.625 - type: nauc_ndcg_at_1000_std value: 40.7925 - type: nauc_ndcg_at_1000_diff1 value: 6.4647 - type: nauc_map_at_1_max value: 43.7806 - type: nauc_map_at_1_std value: 36.2659 - type: nauc_map_at_1_diff1 value: 15.229999999999999 - type: nauc_map_at_3_max value: 46.9934 - type: nauc_map_at_3_std value: 40.6798 - type: nauc_map_at_3_diff1 value: 13.0519 - type: nauc_map_at_5_max value: 44.4704 - type: nauc_map_at_5_std value: 38.4819 - type: nauc_map_at_5_diff1 value: 11.0733 - type: nauc_map_at_10_max value: 45.6016 - type: nauc_map_at_10_std value: 39.923500000000004 - type: nauc_map_at_10_diff1 value: 10.5851 - type: nauc_map_at_20_max value: 45.1017 - type: nauc_map_at_20_std value: 39.647 - type: nauc_map_at_20_diff1 value: 9.8293 - type: nauc_map_at_100_max value: 45.543299999999995 - type: nauc_map_at_100_std value: 40.092 - type: nauc_map_at_100_diff1 value: 9.138499999999999 - type: nauc_map_at_1000_max value: 45.469100000000005 - type: nauc_map_at_1000_std value: 39.9542 - type: nauc_map_at_1000_diff1 value: 9.3956 - type: nauc_recall_at_1_max value: 43.7806 - type: nauc_recall_at_1_std value: 36.2659 - type: nauc_recall_at_1_diff1 value: 15.229999999999999 - type: nauc_recall_at_3_max value: 49.8285 - type: nauc_recall_at_3_std value: 44.6841 - type: nauc_recall_at_3_diff1 value: 10.2662 - type: nauc_recall_at_5_max value: 39.902 - type: nauc_recall_at_5_std value: 35.6058 - type: nauc_recall_at_5_diff1 value: 4.0486 - type: nauc_recall_at_10_max value: 46.808899999999994 - type: nauc_recall_at_10_std value: 43.6669 - type: nauc_recall_at_10_diff1 value: 2.7587 - type: nauc_recall_at_20_max value: 42.3179 - type: nauc_recall_at_20_std value: 41.8701 - type: nauc_recall_at_20_diff1 value: -5.1555 - type: nauc_recall_at_100_max value: 55.4527 - type: nauc_recall_at_100_std value: 57.0136 - type: nauc_recall_at_100_diff1 value: -23.4825 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: 43.7806 - type: nauc_precision_at_1_std value: 36.2659 - type: nauc_precision_at_1_diff1 value: 15.229999999999999 - type: nauc_precision_at_3_max value: 49.8285 - type: nauc_precision_at_3_std value: 44.6841 - type: nauc_precision_at_3_diff1 value: 10.2662 - type: nauc_precision_at_5_max value: 39.902 - type: nauc_precision_at_5_std value: 35.6058 - type: nauc_precision_at_5_diff1 value: 4.0486 - type: nauc_precision_at_10_max value: 46.808899999999994 - type: nauc_precision_at_10_std value: 43.6669 - type: nauc_precision_at_10_diff1 value: 2.7587 - type: nauc_precision_at_20_max value: 42.3179 - type: nauc_precision_at_20_std value: 41.8701 - type: nauc_precision_at_20_diff1 value: -5.1555 - type: nauc_precision_at_100_max value: 55.4527 - type: nauc_precision_at_100_std value: 57.0136 - type: nauc_precision_at_100_diff1 value: -23.4825 - type: nauc_precision_at_1000_max value: 100.0 - type: nauc_precision_at_1000_std value: 100.0 - type: nauc_precision_at_1000_diff1 value: 100.0 - type: nauc_mrr_at_1_max value: 43.7806 - type: nauc_mrr_at_1_std value: 36.2659 - type: nauc_mrr_at_1_diff1 value: 15.229999999999999 - type: nauc_mrr_at_3_max value: 46.9934 - type: nauc_mrr_at_3_std value: 40.6798 - type: nauc_mrr_at_3_diff1 value: 13.0519 - type: nauc_mrr_at_5_max value: 44.4704 - type: nauc_mrr_at_5_std value: 38.4819 - type: nauc_mrr_at_5_diff1 value: 11.0733 - type: nauc_mrr_at_10_max value: 45.6016 - type: nauc_mrr_at_10_std value: 39.923500000000004 - type: nauc_mrr_at_10_diff1 value: 10.5851 - type: nauc_mrr_at_20_max value: 45.1017 - type: nauc_mrr_at_20_std value: 39.647 - type: nauc_mrr_at_20_diff1 value: 9.8293 - type: nauc_mrr_at_100_max value: 45.543299999999995 - type: nauc_mrr_at_100_std value: 40.092 - type: nauc_mrr_at_100_diff1 value: 9.138499999999999 - type: nauc_mrr_at_1000_max value: 45.469100000000005 - type: nauc_mrr_at_1000_std value: 39.9542 - type: nauc_mrr_at_1000_diff1 value: 9.3956 - type: main_score value: 16.588 task: type: Retrieval - dataset: config: ara-zho name: MTEB MLQARetrieval (ara-zho) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 1.5959999999999999 - type: ndcg_at_3 value: 3.2039999999999997 - type: ndcg_at_5 value: 3.868 - type: ndcg_at_10 value: 5.055 - type: ndcg_at_20 value: 6.378 - type: ndcg_at_100 value: 15.976 - type: ndcg_at_1000 value: 20.225 - type: map_at_1 value: 1.5959999999999999 - type: map_at_3 value: 2.8369999999999997 - type: map_at_5 value: 3.209 - type: map_at_10 value: 3.689 - type: map_at_20 value: 4.041 - type: map_at_100 value: 5.132 - type: map_at_1000 value: 5.366 - type: recall_at_1 value: 1.5959999999999999 - type: recall_at_3 value: 4.255 - type: recall_at_5 value: 5.851 - type: recall_at_10 value: 9.574 - type: recall_at_20 value: 14.893999999999998 - type: recall_at_100 value: 70.21300000000001 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 1.5959999999999999 - type: precision_at_3 value: 1.418 - type: precision_at_5 value: 1.17 - type: precision_at_10 value: 0.9570000000000001 - type: precision_at_20 value: 0.745 - type: precision_at_100 value: 0.7020000000000001 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 1.5957 - type: mrr_at_3 value: 2.8369 - type: mrr_at_5 value: 3.2092 - type: mrr_at_10 value: 3.6892 - type: mrr_at_20 value: 4.0411 - type: mrr_at_100 value: 5.1324000000000005 - type: mrr_at_1000 value: 5.3659 - type: nauc_ndcg_at_1_max value: 69.9754 - type: nauc_ndcg_at_1_std value: 49.9288 - type: nauc_ndcg_at_1_diff1 value: 69.9754 - type: nauc_ndcg_at_3_max value: 50.9889 - type: nauc_ndcg_at_3_std value: 50.2248 - type: nauc_ndcg_at_3_diff1 value: 58.6857 - type: nauc_ndcg_at_5_max value: 51.0663 - type: nauc_ndcg_at_5_std value: 50.4333 - type: nauc_ndcg_at_5_diff1 value: 51.5463 - type: nauc_ndcg_at_10_max value: 38.3287 - type: nauc_ndcg_at_10_std value: 32.9963 - type: nauc_ndcg_at_10_diff1 value: 35.6283 - type: nauc_ndcg_at_20_max value: 34.1008 - type: nauc_ndcg_at_20_std value: 30.6261 - type: nauc_ndcg_at_20_diff1 value: 30.673099999999998 - type: nauc_ndcg_at_100_max value: 28.6843 - type: nauc_ndcg_at_100_std value: 22.967599999999997 - type: nauc_ndcg_at_100_diff1 value: 24.8014 - type: nauc_ndcg_at_1000_max value: 36.1468 - type: nauc_ndcg_at_1000_std value: 32.515699999999995 - type: nauc_ndcg_at_1000_diff1 value: 33.9594 - type: nauc_map_at_1_max value: 69.9754 - type: nauc_map_at_1_std value: 49.9288 - type: nauc_map_at_1_diff1 value: 69.9754 - type: nauc_map_at_3_max value: 53.0238 - type: nauc_map_at_3_std value: 50.3964 - type: nauc_map_at_3_diff1 value: 59.9129 - type: nauc_map_at_5_max value: 52.6423 - type: nauc_map_at_5_std value: 50.3198 - type: nauc_map_at_5_diff1 value: 54.6072 - type: nauc_map_at_10_max value: 44.9295 - type: nauc_map_at_10_std value: 40.7116 - type: nauc_map_at_10_diff1 value: 44.9844 - type: nauc_map_at_20_max value: 43.0577 - type: nauc_map_at_20_std value: 39.4453 - type: nauc_map_at_20_diff1 value: 42.8133 - type: nauc_map_at_100_max value: 41.2718 - type: nauc_map_at_100_std value: 37.4294 - type: nauc_map_at_100_diff1 value: 40.6427 - type: nauc_map_at_1000_max value: 42.098400000000005 - type: nauc_map_at_1000_std value: 38.4741 - type: nauc_map_at_1000_diff1 value: 41.6681 - type: nauc_recall_at_1_max value: 69.9754 - type: nauc_recall_at_1_std value: 49.9288 - type: nauc_recall_at_1_diff1 value: 69.9754 - type: nauc_recall_at_3_max value: 47.2231 - type: nauc_recall_at_3_std value: 49.8363 - type: nauc_recall_at_3_diff1 value: 56.4086 - type: nauc_recall_at_5_max value: 48.8223 - type: nauc_recall_at_5_std value: 50.7228 - type: nauc_recall_at_5_diff1 value: 46.4529 - type: nauc_recall_at_10_max value: 29.535899999999998 - type: nauc_recall_at_10_std value: 21.6252 - type: nauc_recall_at_10_diff1 value: 22.9319 - type: nauc_recall_at_20_max value: 23.786099999999998 - type: nauc_recall_at_20_std value: 20.1327 - type: nauc_recall_at_20_diff1 value: 16.564400000000003 - type: nauc_recall_at_100_max value: 13.126299999999999 - type: nauc_recall_at_100_std value: 3.3701000000000003 - type: nauc_recall_at_100_diff1 value: 5.6935 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: 69.9754 - type: nauc_precision_at_1_std value: 49.9288 - type: nauc_precision_at_1_diff1 value: 69.9754 - type: nauc_precision_at_3_max value: 47.2231 - type: nauc_precision_at_3_std value: 49.8363 - type: nauc_precision_at_3_diff1 value: 56.4086 - type: nauc_precision_at_5_max value: 48.8223 - type: nauc_precision_at_5_std value: 50.7228 - type: nauc_precision_at_5_diff1 value: 46.4529 - type: nauc_precision_at_10_max value: 29.535899999999998 - type: nauc_precision_at_10_std value: 21.6252 - type: nauc_precision_at_10_diff1 value: 22.9319 - type: nauc_precision_at_20_max value: 23.786099999999998 - type: nauc_precision_at_20_std value: 20.1327 - type: nauc_precision_at_20_diff1 value: 16.564400000000003 - type: nauc_precision_at_100_max value: 13.126299999999999 - type: nauc_precision_at_100_std value: 3.3701000000000003 - type: nauc_precision_at_100_diff1 value: 5.6935 - type: nauc_precision_at_1000_max value: .nan - type: nauc_precision_at_1000_std value: .nan - type: nauc_precision_at_1000_diff1 value: .nan - type: nauc_mrr_at_1_max value: 69.9754 - type: nauc_mrr_at_1_std value: 49.9288 - type: nauc_mrr_at_1_diff1 value: 69.9754 - type: nauc_mrr_at_3_max value: 53.0238 - type: nauc_mrr_at_3_std value: 50.3964 - type: nauc_mrr_at_3_diff1 value: 59.9129 - type: nauc_mrr_at_5_max value: 52.6423 - type: nauc_mrr_at_5_std value: 50.3198 - type: nauc_mrr_at_5_diff1 value: 54.6072 - type: nauc_mrr_at_10_max value: 44.9295 - type: nauc_mrr_at_10_std value: 40.7116 - type: nauc_mrr_at_10_diff1 value: 44.9844 - type: nauc_mrr_at_20_max value: 43.0577 - type: nauc_mrr_at_20_std value: 39.4453 - type: nauc_mrr_at_20_diff1 value: 42.8133 - type: nauc_mrr_at_100_max value: 41.2718 - type: nauc_mrr_at_100_std value: 37.4294 - type: nauc_mrr_at_100_diff1 value: 40.6427 - type: nauc_mrr_at_1000_max value: 42.098400000000005 - type: nauc_mrr_at_1000_std value: 38.4741 - type: nauc_mrr_at_1000_diff1 value: 41.6681 - type: main_score value: 5.055 task: type: Retrieval - dataset: config: deu-ara name: MTEB MLQARetrieval (deu-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 14.01 - type: ndcg_at_3 value: 18.317 - type: ndcg_at_5 value: 20.334 - type: ndcg_at_10 value: 24.397 - type: ndcg_at_20 value: 26.833000000000002 - type: ndcg_at_100 value: 33.534000000000006 - type: ndcg_at_1000 value: 35.528 - type: map_at_1 value: 14.01 - type: map_at_3 value: 17.150000000000002 - type: map_at_5 value: 18.285 - type: map_at_10 value: 19.962 - type: map_at_20 value: 20.628 - type: map_at_100 value: 21.487000000000002 - type: map_at_1000 value: 21.597 - type: recall_at_1 value: 14.01 - type: recall_at_3 value: 21.739 - type: recall_at_5 value: 26.57 - type: recall_at_10 value: 39.129999999999995 - type: recall_at_20 value: 48.792 - type: recall_at_100 value: 85.99 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 14.01 - type: precision_at_3 value: 7.2459999999999996 - type: precision_at_5 value: 5.314 - type: precision_at_10 value: 3.913 - type: precision_at_20 value: 2.44 - type: precision_at_100 value: 0.86 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 14.0097 - type: mrr_at_3 value: 17.149800000000003 - type: mrr_at_5 value: 18.285 - type: mrr_at_10 value: 19.9617 - type: mrr_at_20 value: 20.627599999999997 - type: mrr_at_100 value: 21.4873 - type: mrr_at_1000 value: 21.5967 - type: nauc_ndcg_at_1_max value: 48.6537 - type: nauc_ndcg_at_1_std value: 43.6024 - type: nauc_ndcg_at_1_diff1 value: 53.921600000000005 - type: nauc_ndcg_at_3_max value: 42.1094 - type: nauc_ndcg_at_3_std value: 38.6143 - type: nauc_ndcg_at_3_diff1 value: 48.1514 - type: nauc_ndcg_at_5_max value: 40.9416 - type: nauc_ndcg_at_5_std value: 37.2043 - type: nauc_ndcg_at_5_diff1 value: 45.0079 - type: nauc_ndcg_at_10_max value: 40.7269 - type: nauc_ndcg_at_10_std value: 36.943 - type: nauc_ndcg_at_10_diff1 value: 39.6211 - type: nauc_ndcg_at_20_max value: 40.2033 - type: nauc_ndcg_at_20_std value: 36.410399999999996 - type: nauc_ndcg_at_20_diff1 value: 36.7802 - type: nauc_ndcg_at_100_max value: 42.459799999999994 - type: nauc_ndcg_at_100_std value: 39.8256 - type: nauc_ndcg_at_100_diff1 value: 38.7772 - type: nauc_ndcg_at_1000_max value: 41.583 - type: nauc_ndcg_at_1000_std value: 38.2014 - type: nauc_ndcg_at_1000_diff1 value: 40.804 - type: nauc_map_at_1_max value: 48.6537 - type: nauc_map_at_1_std value: 43.6024 - type: nauc_map_at_1_diff1 value: 53.921600000000005 - type: nauc_map_at_3_max value: 43.829299999999996 - type: nauc_map_at_3_std value: 39.7968 - type: nauc_map_at_3_diff1 value: 49.2273 - type: nauc_map_at_5_max value: 43.0706 - type: nauc_map_at_5_std value: 38.905699999999996 - type: nauc_map_at_5_diff1 value: 47.344 - type: nauc_map_at_10_max value: 42.7014 - type: nauc_map_at_10_std value: 38.6102 - type: nauc_map_at_10_diff1 value: 44.607400000000005 - type: nauc_map_at_20_max value: 42.5654 - type: nauc_map_at_20_std value: 38.471 - type: nauc_map_at_20_diff1 value: 43.836999999999996 - type: nauc_map_at_100_max value: 42.7715 - type: nauc_map_at_100_std value: 38.905 - type: nauc_map_at_100_diff1 value: 43.985800000000005 - type: nauc_map_at_1000_max value: 42.7228 - type: nauc_map_at_1000_std value: 38.8202 - type: nauc_map_at_1000_diff1 value: 44.0874 - type: nauc_recall_at_1_max value: 48.6537 - type: nauc_recall_at_1_std value: 43.6024 - type: nauc_recall_at_1_diff1 value: 53.921600000000005 - type: nauc_recall_at_3_max value: 37.7806 - type: nauc_recall_at_3_std value: 35.6744 - type: nauc_recall_at_3_diff1 value: 45.5668 - type: nauc_recall_at_5_max value: 35.828500000000005 - type: nauc_recall_at_5_std value: 33.1162 - type: nauc_recall_at_5_diff1 value: 39.3185 - type: nauc_recall_at_10_max value: 36.6766 - type: nauc_recall_at_10_std value: 33.4186 - type: nauc_recall_at_10_diff1 value: 27.6452 - type: nauc_recall_at_20_max value: 34.798 - type: nauc_recall_at_20_std value: 31.506 - type: nauc_recall_at_20_diff1 value: 17.857400000000002 - type: nauc_recall_at_100_max value: 52.4809 - type: nauc_recall_at_100_std value: 59.0823 - type: nauc_recall_at_100_diff1 value: 13.503799999999998 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: 48.6537 - type: nauc_precision_at_1_std value: 43.6024 - type: nauc_precision_at_1_diff1 value: 53.921600000000005 - type: nauc_precision_at_3_max value: 37.7806 - type: nauc_precision_at_3_std value: 35.6744 - type: nauc_precision_at_3_diff1 value: 45.5668 - type: nauc_precision_at_5_max value: 35.828500000000005 - type: nauc_precision_at_5_std value: 33.1162 - type: nauc_precision_at_5_diff1 value: 39.3185 - type: nauc_precision_at_10_max value: 36.6766 - type: nauc_precision_at_10_std value: 33.4186 - type: nauc_precision_at_10_diff1 value: 27.6452 - type: nauc_precision_at_20_max value: 34.798 - type: nauc_precision_at_20_std value: 31.506 - type: nauc_precision_at_20_diff1 value: 17.857400000000002 - type: nauc_precision_at_100_max value: 52.4809 - type: nauc_precision_at_100_std value: 59.0823 - type: nauc_precision_at_100_diff1 value: 13.503799999999998 - type: nauc_precision_at_1000_max value: 100.0 - type: nauc_precision_at_1000_std value: 100.0 - type: nauc_precision_at_1000_diff1 value: 100.0 - type: nauc_mrr_at_1_max value: 48.6537 - type: nauc_mrr_at_1_std value: 43.6024 - type: nauc_mrr_at_1_diff1 value: 53.921600000000005 - type: nauc_mrr_at_3_max value: 43.829299999999996 - type: nauc_mrr_at_3_std value: 39.7968 - type: nauc_mrr_at_3_diff1 value: 49.2273 - type: nauc_mrr_at_5_max value: 43.0706 - type: nauc_mrr_at_5_std value: 38.905699999999996 - type: nauc_mrr_at_5_diff1 value: 47.344 - type: nauc_mrr_at_10_max value: 42.7014 - type: nauc_mrr_at_10_std value: 38.6102 - type: nauc_mrr_at_10_diff1 value: 44.607400000000005 - type: nauc_mrr_at_20_max value: 42.5654 - type: nauc_mrr_at_20_std value: 38.471 - type: nauc_mrr_at_20_diff1 value: 43.836999999999996 - type: nauc_mrr_at_100_max value: 42.7715 - type: nauc_mrr_at_100_std value: 38.905 - type: nauc_mrr_at_100_diff1 value: 43.985800000000005 - type: nauc_mrr_at_1000_max value: 42.7228 - type: nauc_mrr_at_1000_std value: 38.8202 - type: nauc_mrr_at_1000_diff1 value: 44.0874 - type: main_score value: 24.397 task: type: Retrieval - dataset: config: eng-ara name: MTEB MLQARetrieval (eng-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 20.503 - type: ndcg_at_3 value: 28.072000000000003 - type: ndcg_at_5 value: 31.226 - type: ndcg_at_10 value: 33.855000000000004 - type: ndcg_at_20 value: 35.829 - type: ndcg_at_100 value: 40.406 - type: ndcg_at_1000 value: 42.827 - type: map_at_1 value: 20.503 - type: map_at_3 value: 26.209 - type: map_at_5 value: 27.939999999999998 - type: map_at_10 value: 29.026000000000003 - type: map_at_20 value: 29.578 - type: map_at_100 value: 30.203999999999997 - type: map_at_1000 value: 30.309 - type: recall_at_1 value: 20.503 - type: recall_at_3 value: 33.461999999999996 - type: recall_at_5 value: 41.199000000000005 - type: recall_at_10 value: 49.323 - type: recall_at_20 value: 57.06 - type: recall_at_100 value: 81.818 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 20.503 - type: precision_at_3 value: 11.154 - type: precision_at_5 value: 8.24 - type: precision_at_10 value: 4.932 - type: precision_at_20 value: 2.8529999999999998 - type: precision_at_100 value: 0.818 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 20.5029 - type: mrr_at_3 value: 26.2089 - type: mrr_at_5 value: 27.939999999999998 - type: mrr_at_10 value: 29.026000000000003 - type: mrr_at_20 value: 29.578 - type: mrr_at_100 value: 30.204199999999997 - type: mrr_at_1000 value: 30.309399999999997 - type: nauc_ndcg_at_1_max value: 39.9899 - type: nauc_ndcg_at_1_std value: 17.227700000000002 - type: nauc_ndcg_at_1_diff1 value: 53.6614 - type: nauc_ndcg_at_3_max value: 36.1941 - type: nauc_ndcg_at_3_std value: 16.0823 - type: nauc_ndcg_at_3_diff1 value: 41.4409 - type: nauc_ndcg_at_5_max value: 36.227199999999996 - type: nauc_ndcg_at_5_std value: 16.491 - type: nauc_ndcg_at_5_diff1 value: 39.245799999999996 - type: nauc_ndcg_at_10_max value: 35.7319 - type: nauc_ndcg_at_10_std value: 15.437600000000002 - type: nauc_ndcg_at_10_diff1 value: 39.1049 - type: nauc_ndcg_at_20_max value: 36.3035 - type: nauc_ndcg_at_20_std value: 15.3421 - type: nauc_ndcg_at_20_diff1 value: 39.0327 - type: nauc_ndcg_at_100_max value: 36.1578 - type: nauc_ndcg_at_100_std value: 16.5455 - type: nauc_ndcg_at_100_diff1 value: 37.826100000000004 - type: nauc_ndcg_at_1000_max value: 36.5427 - type: nauc_ndcg_at_1000_std value: 16.1092 - type: nauc_ndcg_at_1000_diff1 value: 40.294200000000004 - type: nauc_map_at_1_max value: 39.9899 - type: nauc_map_at_1_std value: 17.227700000000002 - type: nauc_map_at_1_diff1 value: 53.6614 - type: nauc_map_at_3_max value: 37.089800000000004 - type: nauc_map_at_3_std value: 16.3079 - type: nauc_map_at_3_diff1 value: 43.9636 - type: nauc_map_at_5_max value: 37.089800000000004 - type: nauc_map_at_5_std value: 16.5211 - type: nauc_map_at_5_diff1 value: 42.6779 - type: nauc_map_at_10_max value: 36.944300000000005 - type: nauc_map_at_10_std value: 16.0994 - type: nauc_map_at_10_diff1 value: 42.7041 - type: nauc_map_at_20_max value: 37.120999999999995 - type: nauc_map_at_20_std value: 16.062199999999997 - type: nauc_map_at_20_diff1 value: 42.7158 - type: nauc_map_at_100_max value: 37.0936 - type: nauc_map_at_100_std value: 16.2394 - type: nauc_map_at_100_diff1 value: 42.5637 - type: nauc_map_at_1000_max value: 37.1026 - type: nauc_map_at_1000_std value: 16.2146 - type: nauc_map_at_1000_diff1 value: 42.6614 - type: nauc_recall_at_1_max value: 39.9899 - type: nauc_recall_at_1_std value: 17.227700000000002 - type: nauc_recall_at_1_diff1 value: 53.6614 - type: nauc_recall_at_3_max value: 33.820499999999996 - type: nauc_recall_at_3_std value: 15.5002 - type: nauc_recall_at_3_diff1 value: 34.882600000000004 - type: nauc_recall_at_5_max value: 34.0057 - type: nauc_recall_at_5_std value: 16.4978 - type: nauc_recall_at_5_diff1 value: 30.2068 - type: nauc_recall_at_10_max value: 32.218599999999995 - type: nauc_recall_at_10_std value: 13.330300000000001 - type: nauc_recall_at_10_diff1 value: 29.2022 - type: nauc_recall_at_20_max value: 34.082 - type: nauc_recall_at_20_std value: 12.8821 - type: nauc_recall_at_20_diff1 value: 28.0347 - type: nauc_recall_at_100_max value: 31.0029 - type: nauc_recall_at_100_std value: 21.2337 - type: nauc_recall_at_100_diff1 value: 7.781200000000001 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: 39.9899 - type: nauc_precision_at_1_std value: 17.227700000000002 - type: nauc_precision_at_1_diff1 value: 53.6614 - type: nauc_precision_at_3_max value: 33.820499999999996 - type: nauc_precision_at_3_std value: 15.5002 - type: nauc_precision_at_3_diff1 value: 34.882600000000004 - type: nauc_precision_at_5_max value: 34.0057 - type: nauc_precision_at_5_std value: 16.4978 - type: nauc_precision_at_5_diff1 value: 30.2068 - type: nauc_precision_at_10_max value: 32.218599999999995 - type: nauc_precision_at_10_std value: 13.330300000000001 - type: nauc_precision_at_10_diff1 value: 29.2022 - type: nauc_precision_at_20_max value: 34.082 - type: nauc_precision_at_20_std value: 12.8821 - type: nauc_precision_at_20_diff1 value: 28.0347 - type: nauc_precision_at_100_max value: 31.0029 - type: nauc_precision_at_100_std value: 21.2337 - type: nauc_precision_at_100_diff1 value: 7.781200000000001 - type: nauc_precision_at_1000_max value: 100.0 - type: nauc_precision_at_1000_std value: 100.0 - type: nauc_precision_at_1000_diff1 value: 100.0 - type: nauc_mrr_at_1_max value: 39.9899 - type: nauc_mrr_at_1_std value: 17.227700000000002 - type: nauc_mrr_at_1_diff1 value: 53.6614 - type: nauc_mrr_at_3_max value: 37.089800000000004 - type: nauc_mrr_at_3_std value: 16.3079 - type: nauc_mrr_at_3_diff1 value: 43.9636 - type: nauc_mrr_at_5_max value: 37.089800000000004 - type: nauc_mrr_at_5_std value: 16.5211 - type: nauc_mrr_at_5_diff1 value: 42.6779 - type: nauc_mrr_at_10_max value: 36.944300000000005 - type: nauc_mrr_at_10_std value: 16.0994 - type: nauc_mrr_at_10_diff1 value: 42.7041 - type: nauc_mrr_at_20_max value: 37.120999999999995 - type: nauc_mrr_at_20_std value: 16.062199999999997 - type: nauc_mrr_at_20_diff1 value: 42.7158 - type: nauc_mrr_at_100_max value: 37.0936 - type: nauc_mrr_at_100_std value: 16.2394 - type: nauc_mrr_at_100_diff1 value: 42.5637 - type: nauc_mrr_at_1000_max value: 37.1026 - type: nauc_mrr_at_1000_std value: 16.2146 - type: nauc_mrr_at_1000_diff1 value: 42.6614 - type: main_score value: 33.855000000000004 task: type: Retrieval - dataset: config: spa-ara name: MTEB MLQARetrieval (spa-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 14.907 - type: ndcg_at_3 value: 17.717 - type: ndcg_at_5 value: 19.988 - type: ndcg_at_10 value: 22.664 - type: ndcg_at_20 value: 25.235999999999997 - type: ndcg_at_100 value: 33.562999999999995 - type: ndcg_at_1000 value: 34.891 - type: map_at_1 value: 14.907 - type: map_at_3 value: 16.977 - type: map_at_5 value: 18.218999999999998 - type: map_at_10 value: 19.362 - type: map_at_20 value: 20.099 - type: map_at_100 value: 21.096999999999998 - type: map_at_1000 value: 21.17 - type: recall_at_1 value: 14.907 - type: recall_at_3 value: 19.875999999999998 - type: recall_at_5 value: 25.466 - type: recall_at_10 value: 33.54 - type: recall_at_20 value: 43.478 - type: recall_at_100 value: 90.683 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 14.907 - type: precision_at_3 value: 6.625 - type: precision_at_5 value: 5.093 - type: precision_at_10 value: 3.354 - type: precision_at_20 value: 2.174 - type: precision_at_100 value: 0.907 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 14.9068 - type: mrr_at_3 value: 16.9772 - type: mrr_at_5 value: 18.2195 - type: mrr_at_10 value: 19.362399999999997 - type: mrr_at_20 value: 20.0993 - type: mrr_at_100 value: 21.0974 - type: mrr_at_1000 value: 21.1699 - type: nauc_ndcg_at_1_max value: 55.2006 - type: nauc_ndcg_at_1_std value: 40.385799999999996 - type: nauc_ndcg_at_1_diff1 value: 59.772000000000006 - type: nauc_ndcg_at_3_max value: 44.9358 - type: nauc_ndcg_at_3_std value: 32.8022 - type: nauc_ndcg_at_3_diff1 value: 46.041199999999996 - type: nauc_ndcg_at_5_max value: 43.1046 - type: nauc_ndcg_at_5_std value: 33.4372 - type: nauc_ndcg_at_5_diff1 value: 45.1635 - type: nauc_ndcg_at_10_max value: 39.0863 - type: nauc_ndcg_at_10_std value: 32.9405 - type: nauc_ndcg_at_10_diff1 value: 42.248200000000004 - type: nauc_ndcg_at_20_max value: 36.6313 - type: nauc_ndcg_at_20_std value: 30.423299999999998 - type: nauc_ndcg_at_20_diff1 value: 40.221000000000004 - type: nauc_ndcg_at_100_max value: 40.7483 - type: nauc_ndcg_at_100_std value: 32.3215 - type: nauc_ndcg_at_100_diff1 value: 43.728 - type: nauc_ndcg_at_1000_max value: 40.8913 - type: nauc_ndcg_at_1000_std value: 32.4717 - type: nauc_ndcg_at_1000_diff1 value: 44.0186 - type: nauc_map_at_1_max value: 55.2006 - type: nauc_map_at_1_std value: 40.385799999999996 - type: nauc_map_at_1_diff1 value: 59.772000000000006 - type: nauc_map_at_3_max value: 47.3135 - type: nauc_map_at_3_std value: 34.564099999999996 - type: nauc_map_at_3_diff1 value: 49.2551 - type: nauc_map_at_5_max value: 46.0382 - type: nauc_map_at_5_std value: 34.7609 - type: nauc_map_at_5_diff1 value: 48.5978 - type: nauc_map_at_10_max value: 44.115500000000004 - type: nauc_map_at_10_std value: 34.4111 - type: nauc_map_at_10_diff1 value: 47.1515 - type: nauc_map_at_20_max value: 43.3997 - type: nauc_map_at_20_std value: 33.6636 - type: nauc_map_at_20_diff1 value: 46.561099999999996 - type: nauc_map_at_100_max value: 43.7815 - type: nauc_map_at_100_std value: 33.8133 - type: nauc_map_at_100_diff1 value: 46.901399999999995 - type: nauc_map_at_1000_max value: 43.7923 - type: nauc_map_at_1000_std value: 33.8233 - type: nauc_map_at_1000_diff1 value: 46.921099999999996 - type: nauc_recall_at_1_max value: 55.2006 - type: nauc_recall_at_1_std value: 40.385799999999996 - type: nauc_recall_at_1_diff1 value: 59.772000000000006 - type: nauc_recall_at_3_max value: 38.8219 - type: nauc_recall_at_3_std value: 28.269899999999996 - type: nauc_recall_at_3_diff1 value: 37.7657 - type: nauc_recall_at_5_max value: 36.0597 - type: nauc_recall_at_5_std value: 30.4812 - type: nauc_recall_at_5_diff1 value: 36.936600000000006 - type: nauc_recall_at_10_max value: 26.8652 - type: nauc_recall_at_10_std value: 29.919800000000002 - type: nauc_recall_at_10_diff1 value: 30.6267 - type: nauc_recall_at_20_max value: 19.1163 - type: nauc_recall_at_20_std value: 21.9256 - type: nauc_recall_at_20_diff1 value: 24.23 - type: nauc_recall_at_100_max value: 37.845099999999995 - type: nauc_recall_at_100_std value: 29.578 - type: nauc_recall_at_100_diff1 value: 38.3898 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: 55.2006 - type: nauc_precision_at_1_std value: 40.385799999999996 - type: nauc_precision_at_1_diff1 value: 59.772000000000006 - type: nauc_precision_at_3_max value: 38.8219 - type: nauc_precision_at_3_std value: 28.269899999999996 - type: nauc_precision_at_3_diff1 value: 37.7657 - type: nauc_precision_at_5_max value: 36.0597 - type: nauc_precision_at_5_std value: 30.4812 - type: nauc_precision_at_5_diff1 value: 36.936600000000006 - type: nauc_precision_at_10_max value: 26.8652 - type: nauc_precision_at_10_std value: 29.919800000000002 - type: nauc_precision_at_10_diff1 value: 30.6267 - type: nauc_precision_at_20_max value: 19.1163 - type: nauc_precision_at_20_std value: 21.9256 - type: nauc_precision_at_20_diff1 value: 24.23 - type: nauc_precision_at_100_max value: 37.845099999999995 - type: nauc_precision_at_100_std value: 29.578 - type: nauc_precision_at_100_diff1 value: 38.3898 - type: nauc_precision_at_1000_max value: 100.0 - type: nauc_precision_at_1000_std value: 100.0 - type: nauc_precision_at_1000_diff1 value: 100.0 - type: nauc_mrr_at_1_max value: 55.2006 - type: nauc_mrr_at_1_std value: 40.385799999999996 - type: nauc_mrr_at_1_diff1 value: 59.772000000000006 - type: nauc_mrr_at_3_max value: 47.3135 - type: nauc_mrr_at_3_std value: 34.564099999999996 - type: nauc_mrr_at_3_diff1 value: 49.2551 - type: nauc_mrr_at_5_max value: 46.0382 - type: nauc_mrr_at_5_std value: 34.7609 - type: nauc_mrr_at_5_diff1 value: 48.5978 - type: nauc_mrr_at_10_max value: 44.115500000000004 - type: nauc_mrr_at_10_std value: 34.4111 - type: nauc_mrr_at_10_diff1 value: 47.1515 - type: nauc_mrr_at_20_max value: 43.3997 - type: nauc_mrr_at_20_std value: 33.6636 - type: nauc_mrr_at_20_diff1 value: 46.561099999999996 - type: nauc_mrr_at_100_max value: 43.7815 - type: nauc_mrr_at_100_std value: 33.8133 - type: nauc_mrr_at_100_diff1 value: 46.901399999999995 - type: nauc_mrr_at_1000_max value: 43.7923 - type: nauc_mrr_at_1000_std value: 33.8233 - type: nauc_mrr_at_1000_diff1 value: 46.921099999999996 - type: main_score value: 22.664 task: type: Retrieval - dataset: config: hin-ara name: MTEB MLQARetrieval (hin-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 4.839 - type: ndcg_at_3 value: 7.072000000000001 - type: ndcg_at_5 value: 8.623 - type: ndcg_at_10 value: 10.198 - type: ndcg_at_20 value: 12.279 - type: ndcg_at_100 value: 21.737000000000002 - type: ndcg_at_1000 value: 24.826 - type: map_at_1 value: 4.839 - type: map_at_3 value: 6.5409999999999995 - type: map_at_5 value: 7.401000000000001 - type: map_at_10 value: 8.058 - type: map_at_20 value: 8.652 - type: map_at_100 value: 9.817 - type: map_at_1000 value: 9.991999999999999 - type: recall_at_1 value: 4.839 - type: recall_at_3 value: 8.602 - type: recall_at_5 value: 12.366000000000001 - type: recall_at_10 value: 17.204 - type: recall_at_20 value: 25.269000000000002 - type: recall_at_100 value: 78.495 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 4.839 - type: precision_at_3 value: 2.867 - type: precision_at_5 value: 2.473 - type: precision_at_10 value: 1.72 - type: precision_at_20 value: 1.2630000000000001 - type: precision_at_100 value: 0.7849999999999999 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 4.8387 - type: mrr_at_3 value: 6.5412 - type: mrr_at_5 value: 7.4014 - type: mrr_at_10 value: 8.058300000000001 - type: mrr_at_20 value: 8.651499999999999 - type: mrr_at_100 value: 9.8174 - type: mrr_at_1000 value: 9.9919 - type: nauc_ndcg_at_1_max value: 81.63629999999999 - type: nauc_ndcg_at_1_std value: 30.9183 - type: nauc_ndcg_at_1_diff1 value: 88.8087 - type: nauc_ndcg_at_3_max value: 70.5769 - type: nauc_ndcg_at_3_std value: 34.414899999999996 - type: nauc_ndcg_at_3_diff1 value: 58.2588 - type: nauc_ndcg_at_5_max value: 60.49719999999999 - type: nauc_ndcg_at_5_std value: 35.682199999999995 - type: nauc_ndcg_at_5_diff1 value: 55.1545 - type: nauc_ndcg_at_10_max value: 53.2551 - type: nauc_ndcg_at_10_std value: 34.5341 - type: nauc_ndcg_at_10_diff1 value: 46.9268 - type: nauc_ndcg_at_20_max value: 45.9796 - type: nauc_ndcg_at_20_std value: 33.6015 - type: nauc_ndcg_at_20_diff1 value: 38.817 - type: nauc_ndcg_at_100_max value: 44.7291 - type: nauc_ndcg_at_100_std value: 30.532700000000002 - type: nauc_ndcg_at_100_diff1 value: 37.9434 - type: nauc_ndcg_at_1000_max value: 52.945 - type: nauc_ndcg_at_1000_std value: 32.639 - type: nauc_ndcg_at_1000_diff1 value: 45.501999999999995 - type: nauc_map_at_1_max value: 81.63629999999999 - type: nauc_map_at_1_std value: 30.9183 - type: nauc_map_at_1_diff1 value: 88.8087 - type: nauc_map_at_3_max value: 72.8249 - type: nauc_map_at_3_std value: 33.5487 - type: nauc_map_at_3_diff1 value: 63.8737 - type: nauc_map_at_5_max value: 66.1408 - type: nauc_map_at_5_std value: 34.4549 - type: nauc_map_at_5_diff1 value: 61.237399999999994 - type: nauc_map_at_10_max value: 62.1085 - type: nauc_map_at_10_std value: 33.8055 - type: nauc_map_at_10_diff1 value: 56.6972 - type: nauc_map_at_20_max value: 59.1056 - type: nauc_map_at_20_std value: 33.4289 - type: nauc_map_at_20_diff1 value: 53.2797 - type: nauc_map_at_100_max value: 58.9859 - type: nauc_map_at_100_std value: 32.8621 - type: nauc_map_at_100_diff1 value: 52.8525 - type: nauc_map_at_1000_max value: 59.589800000000004 - type: nauc_map_at_1000_std value: 32.9891 - type: nauc_map_at_1000_diff1 value: 53.439899999999994 - type: nauc_recall_at_1_max value: 81.63629999999999 - type: nauc_recall_at_1_std value: 30.9183 - type: nauc_recall_at_1_diff1 value: 88.8087 - type: nauc_recall_at_3_max value: 65.55760000000001 - type: nauc_recall_at_3_std value: 36.392 - type: nauc_recall_at_3_diff1 value: 45.8856 - type: nauc_recall_at_5_max value: 49.4369 - type: nauc_recall_at_5_std value: 38.0439 - type: nauc_recall_at_5_diff1 value: 44.0033 - type: nauc_recall_at_10_max value: 37.8663 - type: nauc_recall_at_10_std value: 35.6881 - type: nauc_recall_at_10_diff1 value: 30.4896 - type: nauc_recall_at_20_max value: 25.0658 - type: nauc_recall_at_20_std value: 33.6279 - type: nauc_recall_at_20_diff1 value: 16.3261 - type: nauc_recall_at_100_max value: 5.8395 - type: nauc_recall_at_100_std value: 20.287399999999998 - type: nauc_recall_at_100_diff1 value: 2.3747000000000003 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: 81.63629999999999 - type: nauc_precision_at_1_std value: 30.9183 - type: nauc_precision_at_1_diff1 value: 88.8087 - type: nauc_precision_at_3_max value: 65.55760000000001 - type: nauc_precision_at_3_std value: 36.392 - type: nauc_precision_at_3_diff1 value: 45.8856 - type: nauc_precision_at_5_max value: 49.4369 - type: nauc_precision_at_5_std value: 38.0439 - type: nauc_precision_at_5_diff1 value: 44.0033 - type: nauc_precision_at_10_max value: 37.8663 - type: nauc_precision_at_10_std value: 35.6881 - type: nauc_precision_at_10_diff1 value: 30.4896 - type: nauc_precision_at_20_max value: 25.0658 - type: nauc_precision_at_20_std value: 33.6279 - type: nauc_precision_at_20_diff1 value: 16.3261 - type: nauc_precision_at_100_max value: 5.8395 - type: nauc_precision_at_100_std value: 20.287399999999998 - type: nauc_precision_at_100_diff1 value: 2.3747000000000003 - type: nauc_precision_at_1000_max value: .nan - type: nauc_precision_at_1000_std value: .nan - type: nauc_precision_at_1000_diff1 value: .nan - type: nauc_mrr_at_1_max value: 81.63629999999999 - type: nauc_mrr_at_1_std value: 30.9183 - type: nauc_mrr_at_1_diff1 value: 88.8087 - type: nauc_mrr_at_3_max value: 72.8249 - type: nauc_mrr_at_3_std value: 33.5487 - type: nauc_mrr_at_3_diff1 value: 63.8737 - type: nauc_mrr_at_5_max value: 66.1408 - type: nauc_mrr_at_5_std value: 34.4549 - type: nauc_mrr_at_5_diff1 value: 61.237399999999994 - type: nauc_mrr_at_10_max value: 62.1085 - type: nauc_mrr_at_10_std value: 33.8055 - type: nauc_mrr_at_10_diff1 value: 56.6972 - type: nauc_mrr_at_20_max value: 59.1056 - type: nauc_mrr_at_20_std value: 33.4289 - type: nauc_mrr_at_20_diff1 value: 53.2797 - type: nauc_mrr_at_100_max value: 58.9859 - type: nauc_mrr_at_100_std value: 32.8621 - type: nauc_mrr_at_100_diff1 value: 52.8525 - type: nauc_mrr_at_1000_max value: 59.589800000000004 - type: nauc_mrr_at_1000_std value: 32.9891 - type: nauc_mrr_at_1000_diff1 value: 53.439899999999994 - type: main_score value: 10.198 task: type: Retrieval - dataset: config: vie-ara name: MTEB MLQARetrieval (vie-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 12.883 - type: ndcg_at_3 value: 18.675 - type: ndcg_at_5 value: 20.945 - type: ndcg_at_10 value: 22.907 - type: ndcg_at_20 value: 25.724000000000004 - type: ndcg_at_100 value: 33.650999999999996 - type: ndcg_at_1000 value: 35.149 - type: map_at_1 value: 12.883 - type: map_at_3 value: 17.28 - type: map_at_5 value: 18.538 - type: map_at_10 value: 19.333 - type: map_at_20 value: 20.119999999999997 - type: map_at_100 value: 21.129 - type: map_at_1000 value: 21.214 - type: recall_at_1 value: 12.883 - type: recall_at_3 value: 22.698999999999998 - type: recall_at_5 value: 28.221 - type: recall_at_10 value: 34.355999999999995 - type: recall_at_20 value: 45.399 - type: recall_at_100 value: 89.571 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 12.883 - type: precision_at_3 value: 7.566000000000001 - type: precision_at_5 value: 5.644 - type: precision_at_10 value: 3.436 - type: precision_at_20 value: 2.27 - type: precision_at_100 value: 0.8959999999999999 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 12.8834 - type: mrr_at_3 value: 17.2802 - type: mrr_at_5 value: 18.537799999999997 - type: mrr_at_10 value: 19.3332 - type: mrr_at_20 value: 20.119799999999998 - type: mrr_at_100 value: 21.129 - type: mrr_at_1000 value: 21.2136 - type: nauc_ndcg_at_1_max value: 54.9133 - type: nauc_ndcg_at_1_std value: 33.8845 - type: nauc_ndcg_at_1_diff1 value: 50.2056 - type: nauc_ndcg_at_3_max value: 52.05610000000001 - type: nauc_ndcg_at_3_std value: 33.2793 - type: nauc_ndcg_at_3_diff1 value: 38.602399999999996 - type: nauc_ndcg_at_5_max value: 50.896300000000004 - type: nauc_ndcg_at_5_std value: 35.9538 - type: nauc_ndcg_at_5_diff1 value: 38.1612 - type: nauc_ndcg_at_10_max value: 49.1237 - type: nauc_ndcg_at_10_std value: 34.8645 - type: nauc_ndcg_at_10_diff1 value: 35.3639 - type: nauc_ndcg_at_20_max value: 50.0941 - type: nauc_ndcg_at_20_std value: 37.4838 - type: nauc_ndcg_at_20_diff1 value: 35.2445 - type: nauc_ndcg_at_100_max value: 49.5562 - type: nauc_ndcg_at_100_std value: 36.1408 - type: nauc_ndcg_at_100_diff1 value: 34.9193 - type: nauc_ndcg_at_1000_max value: 50.476200000000006 - type: nauc_ndcg_at_1000_std value: 36.0392 - type: nauc_ndcg_at_1000_diff1 value: 36.8832 - type: nauc_map_at_1_max value: 54.9133 - type: nauc_map_at_1_std value: 33.8845 - type: nauc_map_at_1_diff1 value: 50.2056 - type: nauc_map_at_3_max value: 52.7482 - type: nauc_map_at_3_std value: 33.173700000000004 - type: nauc_map_at_3_diff1 value: 41.0073 - type: nauc_map_at_5_max value: 52.1539 - type: nauc_map_at_5_std value: 34.8719 - type: nauc_map_at_5_diff1 value: 40.6351 - type: nauc_map_at_10_max value: 51.370599999999996 - type: nauc_map_at_10_std value: 34.458 - type: nauc_map_at_10_diff1 value: 39.4257 - type: nauc_map_at_20_max value: 51.6733 - type: nauc_map_at_20_std value: 35.345 - type: nauc_map_at_20_diff1 value: 39.3542 - type: nauc_map_at_100_max value: 51.477 - type: nauc_map_at_100_std value: 35.185300000000005 - type: nauc_map_at_100_diff1 value: 39.1802 - type: nauc_map_at_1000_max value: 51.53680000000001 - type: nauc_map_at_1000_std value: 35.185300000000005 - type: nauc_map_at_1000_diff1 value: 39.2891 - type: nauc_recall_at_1_max value: 54.9133 - type: nauc_recall_at_1_std value: 33.8845 - type: nauc_recall_at_1_diff1 value: 50.2056 - type: nauc_recall_at_3_max value: 50.3598 - type: nauc_recall_at_3_std value: 33.6083 - type: nauc_recall_at_3_diff1 value: 32.8218 - type: nauc_recall_at_5_max value: 47.8108 - type: nauc_recall_at_5_std value: 38.6411 - type: nauc_recall_at_5_diff1 value: 32.608399999999996 - type: nauc_recall_at_10_max value: 43.4704 - type: nauc_recall_at_10_std value: 35.5433 - type: nauc_recall_at_10_diff1 value: 25.6573 - type: nauc_recall_at_20_max value: 46.5062 - type: nauc_recall_at_20_std value: 43.257 - type: nauc_recall_at_20_diff1 value: 25.5368 - type: nauc_recall_at_100_max value: 36.052099999999996 - type: nauc_recall_at_100_std value: 38.323 - type: nauc_recall_at_100_diff1 value: 4.6289 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: 54.9133 - type: nauc_precision_at_1_std value: 33.8845 - type: nauc_precision_at_1_diff1 value: 50.2056 - type: nauc_precision_at_3_max value: 50.3598 - type: nauc_precision_at_3_std value: 33.6083 - type: nauc_precision_at_3_diff1 value: 32.8218 - type: nauc_precision_at_5_max value: 47.8108 - type: nauc_precision_at_5_std value: 38.6411 - type: nauc_precision_at_5_diff1 value: 32.608399999999996 - type: nauc_precision_at_10_max value: 43.4704 - type: nauc_precision_at_10_std value: 35.5433 - type: nauc_precision_at_10_diff1 value: 25.6573 - type: nauc_precision_at_20_max value: 46.5062 - type: nauc_precision_at_20_std value: 43.257 - type: nauc_precision_at_20_diff1 value: 25.5368 - type: nauc_precision_at_100_max value: 36.052099999999996 - type: nauc_precision_at_100_std value: 38.323 - type: nauc_precision_at_100_diff1 value: 4.6289 - type: nauc_precision_at_1000_max value: 100.0 - type: nauc_precision_at_1000_std value: 100.0 - type: nauc_precision_at_1000_diff1 value: 100.0 - type: nauc_mrr_at_1_max value: 54.9133 - type: nauc_mrr_at_1_std value: 33.8845 - type: nauc_mrr_at_1_diff1 value: 50.2056 - type: nauc_mrr_at_3_max value: 52.7482 - type: nauc_mrr_at_3_std value: 33.173700000000004 - type: nauc_mrr_at_3_diff1 value: 41.0073 - type: nauc_mrr_at_5_max value: 52.1539 - type: nauc_mrr_at_5_std value: 34.8719 - type: nauc_mrr_at_5_diff1 value: 40.6351 - type: nauc_mrr_at_10_max value: 51.370599999999996 - type: nauc_mrr_at_10_std value: 34.458 - type: nauc_mrr_at_10_diff1 value: 39.4257 - type: nauc_mrr_at_20_max value: 51.6733 - type: nauc_mrr_at_20_std value: 35.345 - type: nauc_mrr_at_20_diff1 value: 39.3542 - type: nauc_mrr_at_100_max value: 51.477 - type: nauc_mrr_at_100_std value: 35.185300000000005 - type: nauc_mrr_at_100_diff1 value: 39.1802 - type: nauc_mrr_at_1000_max value: 51.53680000000001 - type: nauc_mrr_at_1000_std value: 35.185300000000005 - type: nauc_mrr_at_1000_diff1 value: 39.2891 - type: main_score value: 22.907 task: type: Retrieval - dataset: config: zho-ara name: MTEB MLQARetrieval (zho-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: validation type: facebook/mlqa metrics: - type: ndcg_at_1 value: 6.915 - type: ndcg_at_3 value: 8.384 - type: ndcg_at_5 value: 10.283000000000001 - type: ndcg_at_10 value: 13.315 - type: ndcg_at_20 value: 15.504999999999999 - type: ndcg_at_100 value: 23.953 - type: ndcg_at_1000 value: 26.767999999999997 - type: map_at_1 value: 6.915 - type: map_at_3 value: 7.979 - type: map_at_5 value: 8.988999999999999 - type: map_at_10 value: 10.204 - type: map_at_20 value: 10.827 - type: map_at_100 value: 11.84 - type: map_at_1000 value: 11.997 - type: recall_at_1 value: 6.915 - type: recall_at_3 value: 9.574 - type: recall_at_5 value: 14.362 - type: recall_at_10 value: 23.936 - type: recall_at_20 value: 32.446999999999996 - type: recall_at_100 value: 80.319 - type: recall_at_1000 value: 100.0 - type: precision_at_1 value: 6.915 - type: precision_at_3 value: 3.1910000000000003 - type: precision_at_5 value: 2.872 - type: precision_at_10 value: 2.394 - type: precision_at_20 value: 1.6219999999999999 - type: precision_at_100 value: 0.803 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 6.9149 - type: mrr_at_3 value: 7.9787 - type: mrr_at_5 value: 8.9894 - type: mrr_at_10 value: 10.2041 - type: mrr_at_20 value: 10.8273 - type: mrr_at_100 value: 11.8397 - type: mrr_at_1000 value: 11.9966 - type: nauc_ndcg_at_1_max value: 51.957 - type: nauc_ndcg_at_1_std value: 12.1946 - type: nauc_ndcg_at_1_diff1 value: 53.8593 - type: nauc_ndcg_at_3_max value: 52.774100000000004 - type: nauc_ndcg_at_3_std value: 18.201800000000002 - type: nauc_ndcg_at_3_diff1 value: 52.7657 - type: nauc_ndcg_at_5_max value: 48.9142 - type: nauc_ndcg_at_5_std value: 16.7029 - type: nauc_ndcg_at_5_diff1 value: 46.0823 - type: nauc_ndcg_at_10_max value: 43.4348 - type: nauc_ndcg_at_10_std value: 20.349600000000002 - type: nauc_ndcg_at_10_diff1 value: 38.9853 - type: nauc_ndcg_at_20_max value: 39.8287 - type: nauc_ndcg_at_20_std value: 18.6698 - type: nauc_ndcg_at_20_diff1 value: 35.056599999999996 - type: nauc_ndcg_at_100_max value: 43.5112 - type: nauc_ndcg_at_100_std value: 18.3445 - type: nauc_ndcg_at_100_diff1 value: 35.4538 - type: nauc_ndcg_at_1000_max value: 44.3752 - type: nauc_ndcg_at_1000_std value: 18.2177 - type: nauc_ndcg_at_1000_diff1 value: 40.2965 - type: nauc_map_at_1_max value: 51.957 - type: nauc_map_at_1_std value: 12.1946 - type: nauc_map_at_1_diff1 value: 53.8593 - type: nauc_map_at_3_max value: 52.8599 - type: nauc_map_at_3_std value: 16.9762 - type: nauc_map_at_3_diff1 value: 53.581500000000005 - type: nauc_map_at_5_max value: 50.3703 - type: nauc_map_at_5_std value: 16.051399999999997 - type: nauc_map_at_5_diff1 value: 49.575599999999994 - type: nauc_map_at_10_max value: 47.7036 - type: nauc_map_at_10_std value: 17.9987 - type: nauc_map_at_10_diff1 value: 45.6932 - type: nauc_map_at_20_max value: 46.3862 - type: nauc_map_at_20_std value: 17.5604 - type: nauc_map_at_20_diff1 value: 44.2286 - type: nauc_map_at_100_max value: 46.9566 - type: nauc_map_at_100_std value: 17.5132 - type: nauc_map_at_100_diff1 value: 44.4521 - type: nauc_map_at_1000_max value: 47.015 - type: nauc_map_at_1000_std value: 17.4815 - type: nauc_map_at_1000_diff1 value: 44.7756 - type: nauc_recall_at_1_max value: 51.957 - type: nauc_recall_at_1_std value: 12.1946 - type: nauc_recall_at_1_diff1 value: 53.8593 - type: nauc_recall_at_3_max value: 52.479299999999995 - type: nauc_recall_at_3_std value: 21.1387 - type: nauc_recall_at_3_diff1 value: 50.6015 - type: nauc_recall_at_5_max value: 45.7547 - type: nauc_recall_at_5_std value: 17.9071 - type: nauc_recall_at_5_diff1 value: 38.3855 - type: nauc_recall_at_10_max value: 35.4748 - type: nauc_recall_at_10_std value: 24.485100000000003 - type: nauc_recall_at_10_diff1 value: 26.963900000000002 - type: nauc_recall_at_20_max value: 27.899400000000004 - type: nauc_recall_at_20_std value: 19.750300000000003 - type: nauc_recall_at_20_diff1 value: 19.0527 - type: nauc_recall_at_100_max value: 37.9463 - type: nauc_recall_at_100_std value: 18.6058 - type: nauc_recall_at_100_diff1 value: 6.7208000000000006 - type: nauc_recall_at_1000_max value: .nan - type: nauc_recall_at_1000_std value: .nan - type: nauc_recall_at_1000_diff1 value: .nan - type: nauc_precision_at_1_max value: 51.957 - type: nauc_precision_at_1_std value: 12.1946 - type: nauc_precision_at_1_diff1 value: 53.8593 - type: nauc_precision_at_3_max value: 52.479299999999995 - type: nauc_precision_at_3_std value: 21.1387 - type: nauc_precision_at_3_diff1 value: 50.6015 - type: nauc_precision_at_5_max value: 45.7547 - type: nauc_precision_at_5_std value: 17.9071 - type: nauc_precision_at_5_diff1 value: 38.3855 - type: nauc_precision_at_10_max value: 35.4748 - type: nauc_precision_at_10_std value: 24.485100000000003 - type: nauc_precision_at_10_diff1 value: 26.963900000000002 - type: nauc_precision_at_20_max value: 27.899400000000004 - type: nauc_precision_at_20_std value: 19.750300000000003 - type: nauc_precision_at_20_diff1 value: 19.0527 - type: nauc_precision_at_100_max value: 37.9463 - type: nauc_precision_at_100_std value: 18.6058 - type: nauc_precision_at_100_diff1 value: 6.7208000000000006 - type: nauc_precision_at_1000_max value: .nan - type: nauc_precision_at_1000_std value: .nan - type: nauc_precision_at_1000_diff1 value: .nan - type: nauc_mrr_at_1_max value: 51.957 - type: nauc_mrr_at_1_std value: 12.1946 - type: nauc_mrr_at_1_diff1 value: 53.8593 - type: nauc_mrr_at_3_max value: 52.8599 - type: nauc_mrr_at_3_std value: 16.9762 - type: nauc_mrr_at_3_diff1 value: 53.581500000000005 - type: nauc_mrr_at_5_max value: 50.3703 - type: nauc_mrr_at_5_std value: 16.051399999999997 - type: nauc_mrr_at_5_diff1 value: 49.575599999999994 - type: nauc_mrr_at_10_max value: 47.7036 - type: nauc_mrr_at_10_std value: 17.9987 - type: nauc_mrr_at_10_diff1 value: 45.6932 - type: nauc_mrr_at_20_max value: 46.3862 - type: nauc_mrr_at_20_std value: 17.5604 - type: nauc_mrr_at_20_diff1 value: 44.2286 - type: nauc_mrr_at_100_max value: 46.9566 - type: nauc_mrr_at_100_std value: 17.5132 - type: nauc_mrr_at_100_diff1 value: 44.4521 - type: nauc_mrr_at_1000_max value: 47.0151 - type: nauc_mrr_at_1000_std value: 17.4815 - type: nauc_mrr_at_1000_diff1 value: 44.7757 - type: main_score value: 13.315 task: type: Retrieval - dataset: config: ara-ara name: MTEB MLQARetrieval (ara-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 39.265 - type: ndcg_at_3 value: 46.804 - type: ndcg_at_5 value: 48.903999999999996 - type: ndcg_at_10 value: 50.926 - type: ndcg_at_20 value: 52.452 - type: ndcg_at_100 value: 54.827000000000005 - type: ndcg_at_1000 value: 56.485 - type: map_at_1 value: 39.256 - type: map_at_3 value: 44.962 - type: map_at_5 value: 46.123 - type: map_at_10 value: 46.958 - type: map_at_20 value: 47.375 - type: map_at_100 value: 47.697 - type: map_at_1000 value: 47.753 - type: recall_at_1 value: 39.256 - type: recall_at_3 value: 52.11900000000001 - type: recall_at_5 value: 57.238 - type: recall_at_10 value: 63.482000000000006 - type: recall_at_20 value: 69.539 - type: recall_at_100 value: 82.43 - type: recall_at_1000 value: 95.837 - type: precision_at_1 value: 39.265 - type: precision_at_3 value: 17.376 - type: precision_at_5 value: 11.449 - type: precision_at_10 value: 6.349 - type: precision_at_20 value: 3.4770000000000003 - type: precision_at_100 value: 0.8240000000000001 - type: precision_at_1000 value: 0.096 - type: mrr_at_1 value: 39.265 - type: mrr_at_3 value: 44.9716 - type: mrr_at_5 value: 46.1323 - type: mrr_at_10 value: 46.9677 - type: mrr_at_20 value: 47.3842 - type: mrr_at_100 value: 47.7064 - type: mrr_at_1000 value: 47.762100000000004 - type: nauc_ndcg_at_1_max value: 61.7787 - type: nauc_ndcg_at_1_std value: 6.1566 - type: nauc_ndcg_at_1_diff1 value: 67.6182 - type: nauc_ndcg_at_3_max value: 63.207899999999995 - type: nauc_ndcg_at_3_std value: 9.4223 - type: nauc_ndcg_at_3_diff1 value: 61.241 - type: nauc_ndcg_at_5_max value: 62.7074 - type: nauc_ndcg_at_5_std value: 10.0744 - type: nauc_ndcg_at_5_diff1 value: 59.9704 - type: nauc_ndcg_at_10_max value: 62.584300000000006 - type: nauc_ndcg_at_10_std value: 11.291 - type: nauc_ndcg_at_10_diff1 value: 59.044700000000006 - type: nauc_ndcg_at_20_max value: 62.5465 - type: nauc_ndcg_at_20_std value: 12.02 - type: nauc_ndcg_at_20_diff1 value: 58.817 - type: nauc_ndcg_at_100_max value: 62.5796 - type: nauc_ndcg_at_100_std value: 12.5578 - type: nauc_ndcg_at_100_diff1 value: 58.812200000000004 - type: nauc_ndcg_at_1000_max value: 62.614599999999996 - type: nauc_ndcg_at_1000_std value: 11.786000000000001 - type: nauc_ndcg_at_1000_diff1 value: 59.748599999999996 - type: nauc_map_at_1_max value: 61.7363 - type: nauc_map_at_1_std value: 6.127 - type: nauc_map_at_1_diff1 value: 67.6168 - type: nauc_map_at_3_max value: 62.8636 - type: nauc_map_at_3_std value: 8.5937 - type: nauc_map_at_3_diff1 value: 62.770199999999996 - type: nauc_map_at_5_max value: 62.583999999999996 - type: nauc_map_at_5_std value: 8.912 - type: nauc_map_at_5_diff1 value: 62.095800000000004 - type: nauc_map_at_10_max value: 62.5306 - type: nauc_map_at_10_std value: 9.3898 - type: nauc_map_at_10_diff1 value: 61.7473 - type: nauc_map_at_20_max value: 62.5139 - type: nauc_map_at_20_std value: 9.5692 - type: nauc_map_at_20_diff1 value: 61.696200000000005 - type: nauc_map_at_100_max value: 62.5226 - type: nauc_map_at_100_std value: 9.6093 - type: nauc_map_at_100_diff1 value: 61.7158 - type: nauc_map_at_1000_max value: 62.522999999999996 - type: nauc_map_at_1000_std value: 9.5925 - type: nauc_map_at_1000_diff1 value: 61.7435 - type: nauc_recall_at_1_max value: 61.7363 - type: nauc_recall_at_1_std value: 6.127 - type: nauc_recall_at_1_diff1 value: 67.6168 - type: nauc_recall_at_3_max value: 64.1748 - type: nauc_recall_at_3_std value: 11.8668 - type: nauc_recall_at_3_diff1 value: 56.694500000000005 - type: nauc_recall_at_5_max value: 62.9549 - type: nauc_recall_at_5_std value: 13.802200000000001 - type: nauc_recall_at_5_diff1 value: 53.172 - type: nauc_recall_at_10_max value: 62.564 - type: nauc_recall_at_10_std value: 18.3922 - type: nauc_recall_at_10_diff1 value: 49.275999999999996 - type: nauc_recall_at_20_max value: 62.440799999999996 - type: nauc_recall_at_20_std value: 22.8633 - type: nauc_recall_at_20_diff1 value: 47.0028 - type: nauc_recall_at_100_max value: 62.5706 - type: nauc_recall_at_100_std value: 34.836099999999995 - type: nauc_recall_at_100_diff1 value: 40.109 - type: nauc_recall_at_1000_max value: 64.8292 - type: nauc_recall_at_1000_std value: 52.729000000000006 - type: nauc_recall_at_1000_diff1 value: 39.4034 - type: nauc_precision_at_1_max value: 61.7787 - type: nauc_precision_at_1_std value: 6.1566 - type: nauc_precision_at_1_diff1 value: 67.6182 - type: nauc_precision_at_3_max value: 64.2287 - type: nauc_precision_at_3_std value: 11.8995 - type: nauc_precision_at_3_diff1 value: 56.704699999999995 - type: nauc_precision_at_5_max value: 63.011700000000005 - type: nauc_precision_at_5_std value: 13.8369 - type: nauc_precision_at_5_diff1 value: 53.1825 - type: nauc_precision_at_10_max value: 62.6302 - type: nauc_precision_at_10_std value: 18.432000000000002 - type: nauc_precision_at_10_diff1 value: 49.2902 - type: nauc_precision_at_20_max value: 62.5155 - type: nauc_precision_at_20_std value: 22.9091 - type: nauc_precision_at_20_diff1 value: 47.0182 - type: nauc_precision_at_100_max value: 62.6993 - type: nauc_precision_at_100_std value: 34.928599999999996 - type: nauc_precision_at_100_diff1 value: 40.1379 - type: nauc_precision_at_1000_max value: 64.35459999999999 - type: nauc_precision_at_1000_std value: 52.3714 - type: nauc_precision_at_1000_diff1 value: 38.7124 - type: nauc_mrr_at_1_max value: 61.7787 - type: nauc_mrr_at_1_std value: 6.1566 - type: nauc_mrr_at_1_diff1 value: 67.6182 - type: nauc_mrr_at_3_max value: 62.9069 - type: nauc_mrr_at_3_std value: 8.6238 - type: nauc_mrr_at_3_diff1 value: 62.7719 - type: nauc_mrr_at_5_max value: 62.6278 - type: nauc_mrr_at_5_std value: 8.9425 - type: nauc_mrr_at_5_diff1 value: 62.097500000000004 - type: nauc_mrr_at_10_max value: 62.5749 - type: nauc_mrr_at_10_std value: 9.4205 - type: nauc_mrr_at_10_diff1 value: 61.7491 - type: nauc_mrr_at_20_max value: 62.558499999999995 - type: nauc_mrr_at_20_std value: 9.600200000000001 - type: nauc_mrr_at_20_diff1 value: 61.698 - type: nauc_mrr_at_100_max value: 62.56739999999999 - type: nauc_mrr_at_100_std value: 9.640500000000001 - type: nauc_mrr_at_100_diff1 value: 61.7177 - type: nauc_mrr_at_1000_max value: 62.56739999999999 - type: nauc_mrr_at_1000_std value: 9.6233 - type: nauc_mrr_at_1000_diff1 value: 61.7453 - type: main_score value: 50.926 task: type: Retrieval - dataset: config: ara-deu name: MTEB MLQARetrieval (ara-deu) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 2.852 - type: ndcg_at_3 value: 4.141 - type: ndcg_at_5 value: 4.789000000000001 - type: ndcg_at_10 value: 5.507 - type: ndcg_at_20 value: 6.4399999999999995 - type: ndcg_at_100 value: 9.126 - type: ndcg_at_1000 value: 15.389 - type: map_at_1 value: 2.852 - type: map_at_3 value: 3.8129999999999997 - type: map_at_5 value: 4.170999999999999 - type: map_at_10 value: 4.473 - type: map_at_20 value: 4.728000000000001 - type: map_at_100 value: 5.045 - type: map_at_1000 value: 5.2139999999999995 - type: recall_at_1 value: 2.852 - type: recall_at_3 value: 5.097 - type: recall_at_5 value: 6.675000000000001 - type: recall_at_10 value: 8.859 - type: recall_at_20 value: 12.561 - type: recall_at_100 value: 27.852 - type: recall_at_1000 value: 81.038 - type: precision_at_1 value: 2.852 - type: precision_at_3 value: 1.699 - type: precision_at_5 value: 1.335 - type: precision_at_10 value: 0.886 - type: precision_at_20 value: 0.628 - type: precision_at_100 value: 0.27899999999999997 - type: precision_at_1000 value: 0.08099999999999999 - type: mrr_at_1 value: 2.8519 - type: mrr_at_3 value: 3.8127 - type: mrr_at_5 value: 4.1707 - type: mrr_at_10 value: 4.473400000000001 - type: mrr_at_20 value: 4.728000000000001 - type: mrr_at_100 value: 5.0449 - type: mrr_at_1000 value: 5.2147 - type: nauc_ndcg_at_1_max value: 23.743100000000002 - type: nauc_ndcg_at_1_std value: -3.2365999999999997 - type: nauc_ndcg_at_1_diff1 value: 42.0576 - type: nauc_ndcg_at_3_max value: 17.3399 - type: nauc_ndcg_at_3_std value: 1.6805 - type: nauc_ndcg_at_3_diff1 value: 28.59 - type: nauc_ndcg_at_5_max value: 14.735599999999998 - type: nauc_ndcg_at_5_std value: 1.6570999999999998 - type: nauc_ndcg_at_5_diff1 value: 26.857300000000002 - type: nauc_ndcg_at_10_max value: 13.1464 - type: nauc_ndcg_at_10_std value: 1.9082 - type: nauc_ndcg_at_10_diff1 value: 25.200499999999998 - type: nauc_ndcg_at_20_max value: 11.1051 - type: nauc_ndcg_at_20_std value: 3.0543 - type: nauc_ndcg_at_20_diff1 value: 21.5362 - type: nauc_ndcg_at_100_max value: 6.245 - type: nauc_ndcg_at_100_std value: 2.4553 - type: nauc_ndcg_at_100_diff1 value: 15.776200000000001 - type: nauc_ndcg_at_1000_max value: 8.6287 - type: nauc_ndcg_at_1000_std value: 2.384 - type: nauc_ndcg_at_1000_diff1 value: 18.2791 - type: nauc_map_at_1_max value: 23.743100000000002 - type: nauc_map_at_1_std value: -3.2365999999999997 - type: nauc_map_at_1_diff1 value: 42.0576 - type: nauc_map_at_3_max value: 18.5911 - type: nauc_map_at_3_std value: 0.8439 - type: nauc_map_at_3_diff1 value: 31.1953 - type: nauc_map_at_5_max value: 16.807 - type: nauc_map_at_5_std value: 0.8949 - type: nauc_map_at_5_diff1 value: 29.8614 - type: nauc_map_at_10_max value: 15.8644 - type: nauc_map_at_10_std value: 1.0688 - type: nauc_map_at_10_diff1 value: 28.8088 - type: nauc_map_at_20_max value: 14.9936 - type: nauc_map_at_20_std value: 1.4885000000000002 - type: nauc_map_at_20_diff1 value: 27.2399 - type: nauc_map_at_100_max value: 13.8854 - type: nauc_map_at_100_std value: 1.4527999999999999 - type: nauc_map_at_100_diff1 value: 25.9019 - type: nauc_map_at_1000_max value: 13.964599999999999 - type: nauc_map_at_1000_std value: 1.4558 - type: nauc_map_at_1000_diff1 value: 26.0032 - type: nauc_recall_at_1_max value: 23.743100000000002 - type: nauc_recall_at_1_std value: -3.2365999999999997 - type: nauc_recall_at_1_diff1 value: 42.0576 - type: nauc_recall_at_3_max value: 14.6093 - type: nauc_recall_at_3_std value: 3.4709 - type: nauc_recall_at_3_diff1 value: 22.912 - type: nauc_recall_at_5_max value: 10.763300000000001 - type: nauc_recall_at_5_std value: 3.0224 - type: nauc_recall_at_5_diff1 value: 21.251800000000003 - type: nauc_recall_at_10_max value: 8.6732 - type: nauc_recall_at_10_std value: 3.1818 - type: nauc_recall_at_10_diff1 value: 19.4694 - type: nauc_recall_at_20_max value: 5.4847 - type: nauc_recall_at_20_std value: 5.3785 - type: nauc_recall_at_20_diff1 value: 13.377 - type: nauc_recall_at_100_max value: -3.0175 - type: nauc_recall_at_100_std value: 3.0078 - type: nauc_recall_at_100_diff1 value: 3.8725 - type: nauc_recall_at_1000_max value: 2.6972 - type: nauc_recall_at_1000_std value: 3.1039 - type: nauc_recall_at_1000_diff1 value: 7.201200000000001 - type: nauc_precision_at_1_max value: 23.743100000000002 - type: nauc_precision_at_1_std value: -3.2365999999999997 - type: nauc_precision_at_1_diff1 value: 42.0576 - type: nauc_precision_at_3_max value: 14.6093 - type: nauc_precision_at_3_std value: 3.4709 - type: nauc_precision_at_3_diff1 value: 22.912 - type: nauc_precision_at_5_max value: 10.763300000000001 - type: nauc_precision_at_5_std value: 3.0224 - type: nauc_precision_at_5_diff1 value: 21.251800000000003 - type: nauc_precision_at_10_max value: 8.6732 - type: nauc_precision_at_10_std value: 3.1818 - type: nauc_precision_at_10_diff1 value: 19.4694 - type: nauc_precision_at_20_max value: 5.4847 - type: nauc_precision_at_20_std value: 5.3785 - type: nauc_precision_at_20_diff1 value: 13.377 - type: nauc_precision_at_100_max value: -3.0175 - type: nauc_precision_at_100_std value: 3.0078 - type: nauc_precision_at_100_diff1 value: 3.8725 - type: nauc_precision_at_1000_max value: 2.5304 - type: nauc_precision_at_1000_std value: 2.9852 - type: nauc_precision_at_1000_diff1 value: 7.1508 - type: nauc_mrr_at_1_max value: 23.743100000000002 - type: nauc_mrr_at_1_std value: -3.2365999999999997 - type: nauc_mrr_at_1_diff1 value: 42.0576 - type: nauc_mrr_at_3_max value: 18.5911 - type: nauc_mrr_at_3_std value: 0.8439 - type: nauc_mrr_at_3_diff1 value: 31.1953 - type: nauc_mrr_at_5_max value: 16.807 - type: nauc_mrr_at_5_std value: 0.8949 - type: nauc_mrr_at_5_diff1 value: 29.8614 - type: nauc_mrr_at_10_max value: 15.8644 - type: nauc_mrr_at_10_std value: 1.0688 - type: nauc_mrr_at_10_diff1 value: 28.8088 - type: nauc_mrr_at_20_max value: 14.9936 - type: nauc_mrr_at_20_std value: 1.4885000000000002 - type: nauc_mrr_at_20_diff1 value: 27.2399 - type: nauc_mrr_at_100_max value: 13.8854 - type: nauc_mrr_at_100_std value: 1.4527999999999999 - type: nauc_mrr_at_100_diff1 value: 25.9019 - type: nauc_mrr_at_1000_max value: 13.961200000000002 - type: nauc_mrr_at_1000_std value: 1.4534 - type: nauc_mrr_at_1000_diff1 value: 26.001800000000003 - type: main_score value: 5.507 task: type: Retrieval - dataset: config: ara-eng name: MTEB MLQARetrieval (ara-eng) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 4.689 - type: ndcg_at_3 value: 6.6979999999999995 - type: ndcg_at_5 value: 7.48 - type: ndcg_at_10 value: 8.559 - type: ndcg_at_20 value: 9.496 - type: ndcg_at_100 value: 11.894 - type: ndcg_at_1000 value: 16.217000000000002 - type: map_at_1 value: 4.689 - type: map_at_3 value: 6.189 - type: map_at_5 value: 6.624 - type: map_at_10 value: 7.066 - type: map_at_20 value: 7.321999999999999 - type: map_at_100 value: 7.625 - type: map_at_1000 value: 7.753 - type: recall_at_1 value: 4.689 - type: recall_at_3 value: 8.177 - type: recall_at_5 value: 10.071 - type: recall_at_10 value: 13.428 - type: recall_at_20 value: 17.142 - type: recall_at_100 value: 30.505 - type: recall_at_1000 value: 66.607 - type: precision_at_1 value: 4.689 - type: precision_at_3 value: 2.726 - type: precision_at_5 value: 2.0140000000000002 - type: precision_at_10 value: 1.343 - type: precision_at_20 value: 0.857 - type: precision_at_100 value: 0.305 - type: precision_at_1000 value: 0.067 - type: mrr_at_1 value: 4.6887 - type: mrr_at_3 value: 6.189 - type: mrr_at_5 value: 6.624199999999999 - type: mrr_at_10 value: 7.0656 - type: mrr_at_20 value: 7.321999999999999 - type: mrr_at_100 value: 7.6248 - type: mrr_at_1000 value: 7.7528 - type: nauc_ndcg_at_1_max value: 51.2103 - type: nauc_ndcg_at_1_std value: 29.4788 - type: nauc_ndcg_at_1_diff1 value: 37.000499999999995 - type: nauc_ndcg_at_3_max value: 46.1515 - type: nauc_ndcg_at_3_std value: 30.2904 - type: nauc_ndcg_at_3_diff1 value: 26.732899999999997 - type: nauc_ndcg_at_5_max value: 43.5112 - type: nauc_ndcg_at_5_std value: 29.5112 - type: nauc_ndcg_at_5_diff1 value: 24.0178 - type: nauc_ndcg_at_10_max value: 41.7748 - type: nauc_ndcg_at_10_std value: 29.709999999999997 - type: nauc_ndcg_at_10_diff1 value: 22.1814 - type: nauc_ndcg_at_20_max value: 40.5753 - type: nauc_ndcg_at_20_std value: 29.6339 - type: nauc_ndcg_at_20_diff1 value: 20.8567 - type: nauc_ndcg_at_100_max value: 38.7065 - type: nauc_ndcg_at_100_std value: 30.197400000000002 - type: nauc_ndcg_at_100_diff1 value: 18.0378 - type: nauc_ndcg_at_1000_max value: 39.0409 - type: nauc_ndcg_at_1000_std value: 31.1176 - type: nauc_ndcg_at_1000_diff1 value: 18.1937 - type: nauc_map_at_1_max value: 51.2103 - type: nauc_map_at_1_std value: 29.4788 - type: nauc_map_at_1_diff1 value: 37.000499999999995 - type: nauc_map_at_3_max value: 47.132400000000004 - type: nauc_map_at_3_std value: 30.067 - type: nauc_map_at_3_diff1 value: 28.6981 - type: nauc_map_at_5_max value: 45.4253 - type: nauc_map_at_5_std value: 29.576 - type: nauc_map_at_5_diff1 value: 26.863999999999997 - type: nauc_map_at_10_max value: 44.5113 - type: nauc_map_at_10_std value: 29.6666 - type: nauc_map_at_10_diff1 value: 25.8586 - type: nauc_map_at_20_max value: 44.0195 - type: nauc_map_at_20_std value: 29.6244 - type: nauc_map_at_20_diff1 value: 25.3006 - type: nauc_map_at_100_max value: 43.621500000000005 - type: nauc_map_at_100_std value: 29.7454 - type: nauc_map_at_100_diff1 value: 24.6802 - type: nauc_map_at_1000_max value: 43.6197 - type: nauc_map_at_1000_std value: 29.781200000000002 - type: nauc_map_at_1000_diff1 value: 24.6832 - type: nauc_recall_at_1_max value: 51.2103 - type: nauc_recall_at_1_std value: 29.4788 - type: nauc_recall_at_1_diff1 value: 37.000499999999995 - type: nauc_recall_at_3_max value: 43.9912 - type: nauc_recall_at_3_std value: 30.801699999999997 - type: nauc_recall_at_3_diff1 value: 22.412399999999998 - type: nauc_recall_at_5_max value: 39.598299999999995 - type: nauc_recall_at_5_std value: 29.3477 - type: nauc_recall_at_5_diff1 value: 18.3309 - type: nauc_recall_at_10_max value: 36.679 - type: nauc_recall_at_10_std value: 29.7989 - type: nauc_recall_at_10_diff1 value: 15.534400000000002 - type: nauc_recall_at_20_max value: 34.7491 - type: nauc_recall_at_20_std value: 29.6655 - type: nauc_recall_at_20_diff1 value: 13.5716 - type: nauc_recall_at_100_max value: 31.2425 - type: nauc_recall_at_100_std value: 31.047200000000004 - type: nauc_recall_at_100_diff1 value: 8.0658 - type: nauc_recall_at_1000_max value: 31.5178 - type: nauc_recall_at_1000_std value: 35.7175 - type: nauc_recall_at_1000_diff1 value: 6.5687999999999995 - type: nauc_precision_at_1_max value: 51.2103 - type: nauc_precision_at_1_std value: 29.4788 - type: nauc_precision_at_1_diff1 value: 37.000499999999995 - type: nauc_precision_at_3_max value: 43.9912 - type: nauc_precision_at_3_std value: 30.801699999999997 - type: nauc_precision_at_3_diff1 value: 22.412399999999998 - type: nauc_precision_at_5_max value: 39.598299999999995 - type: nauc_precision_at_5_std value: 29.3477 - type: nauc_precision_at_5_diff1 value: 18.3309 - type: nauc_precision_at_10_max value: 36.679 - type: nauc_precision_at_10_std value: 29.7989 - type: nauc_precision_at_10_diff1 value: 15.534400000000002 - type: nauc_precision_at_20_max value: 34.7491 - type: nauc_precision_at_20_std value: 29.6655 - type: nauc_precision_at_20_diff1 value: 13.5716 - type: nauc_precision_at_100_max value: 31.2457 - type: nauc_precision_at_100_std value: 31.0504 - type: nauc_precision_at_100_diff1 value: 8.0557 - type: nauc_precision_at_1000_max value: 31.4683 - type: nauc_precision_at_1000_std value: 35.7657 - type: nauc_precision_at_1000_diff1 value: 6.4829 - type: nauc_mrr_at_1_max value: 51.2103 - type: nauc_mrr_at_1_std value: 29.4788 - type: nauc_mrr_at_1_diff1 value: 37.000499999999995 - type: nauc_mrr_at_3_max value: 47.132400000000004 - type: nauc_mrr_at_3_std value: 30.067 - type: nauc_mrr_at_3_diff1 value: 28.6981 - type: nauc_mrr_at_5_max value: 45.4253 - type: nauc_mrr_at_5_std value: 29.576 - type: nauc_mrr_at_5_diff1 value: 26.863999999999997 - type: nauc_mrr_at_10_max value: 44.5113 - type: nauc_mrr_at_10_std value: 29.6666 - type: nauc_mrr_at_10_diff1 value: 25.8586 - type: nauc_mrr_at_20_max value: 44.0195 - type: nauc_mrr_at_20_std value: 29.6244 - type: nauc_mrr_at_20_diff1 value: 25.3006 - type: nauc_mrr_at_100_max value: 43.6216 - type: nauc_mrr_at_100_std value: 29.745500000000003 - type: nauc_mrr_at_100_diff1 value: 24.6799 - type: nauc_mrr_at_1000_max value: 43.6198 - type: nauc_mrr_at_1000_std value: 29.781299999999998 - type: nauc_mrr_at_1000_diff1 value: 24.6829 - type: main_score value: 8.559 task: type: Retrieval - dataset: config: ara-spa name: MTEB MLQARetrieval (ara-spa) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 3.134 - type: ndcg_at_3 value: 4.329000000000001 - type: ndcg_at_5 value: 4.8919999999999995 - type: ndcg_at_10 value: 5.5169999999999995 - type: ndcg_at_20 value: 6.265999999999999 - type: ndcg_at_100 value: 8.331 - type: ndcg_at_1000 value: 14.316999999999998 - type: map_at_1 value: 3.134 - type: map_at_3 value: 4.011 - type: map_at_5 value: 4.324 - type: map_at_10 value: 4.585 - type: map_at_20 value: 4.7940000000000005 - type: map_at_100 value: 5.054 - type: map_at_1000 value: 5.206 - type: recall_at_1 value: 3.134 - type: recall_at_3 value: 5.258 - type: recall_at_5 value: 6.622999999999999 - type: recall_at_10 value: 8.544 - type: recall_at_20 value: 11.476 - type: recall_at_100 value: 23.003 - type: recall_at_1000 value: 74.57000000000001 - type: precision_at_1 value: 3.134 - type: precision_at_3 value: 1.7530000000000001 - type: precision_at_5 value: 1.325 - type: precision_at_10 value: 0.8540000000000001 - type: precision_at_20 value: 0.5740000000000001 - type: precision_at_100 value: 0.22999999999999998 - type: precision_at_1000 value: 0.075 - type: mrr_at_1 value: 3.1344999999999996 - type: mrr_at_3 value: 4.0108 - type: mrr_at_5 value: 4.3242 - type: mrr_at_10 value: 4.5849 - type: mrr_at_20 value: 4.7944 - type: mrr_at_100 value: 5.0545 - type: mrr_at_1000 value: 5.2062 - type: nauc_ndcg_at_1_max value: 37.7857 - type: nauc_ndcg_at_1_std value: 28.0731 - type: nauc_ndcg_at_1_diff1 value: 30.3693 - type: nauc_ndcg_at_3_max value: 31.761499999999998 - type: nauc_ndcg_at_3_std value: 29.881 - type: nauc_ndcg_at_3_diff1 value: 17.308899999999998 - type: nauc_ndcg_at_5_max value: 29.438 - type: nauc_ndcg_at_5_std value: 29.3066 - type: nauc_ndcg_at_5_diff1 value: 15.1944 - type: nauc_ndcg_at_10_max value: 27.798000000000002 - type: nauc_ndcg_at_10_std value: 28.841299999999997 - type: nauc_ndcg_at_10_diff1 value: 12.5614 - type: nauc_ndcg_at_20_max value: 25.982499999999998 - type: nauc_ndcg_at_20_std value: 28.201999999999998 - type: nauc_ndcg_at_20_diff1 value: 11.0868 - type: nauc_ndcg_at_100_max value: 21.0176 - type: nauc_ndcg_at_100_std value: 26.584600000000002 - type: nauc_ndcg_at_100_diff1 value: 7.4644 - type: nauc_ndcg_at_1000_max value: 19.7748 - type: nauc_ndcg_at_1000_std value: 25.3763 - type: nauc_ndcg_at_1000_diff1 value: 6.0342 - type: nauc_map_at_1_max value: 37.7857 - type: nauc_map_at_1_std value: 28.0731 - type: nauc_map_at_1_diff1 value: 30.3693 - type: nauc_map_at_3_max value: 32.8434 - type: nauc_map_at_3_std value: 29.6022 - type: nauc_map_at_3_diff1 value: 19.8538 - type: nauc_map_at_5_max value: 31.306099999999997 - type: nauc_map_at_5_std value: 29.305500000000002 - type: nauc_map_at_5_diff1 value: 18.3086 - type: nauc_map_at_10_max value: 30.3611 - type: nauc_map_at_10_std value: 29.0105 - type: nauc_map_at_10_diff1 value: 16.7745 - type: nauc_map_at_20_max value: 29.5693 - type: nauc_map_at_20_std value: 28.774 - type: nauc_map_at_20_diff1 value: 16.084899999999998 - type: nauc_map_at_100_max value: 28.548099999999998 - type: nauc_map_at_100_std value: 28.522 - type: nauc_map_at_100_diff1 value: 15.317900000000002 - type: nauc_map_at_1000_max value: 28.412399999999998 - type: nauc_map_at_1000_std value: 28.3895 - type: nauc_map_at_1000_diff1 value: 15.209600000000002 - type: nauc_recall_at_1_max value: 37.7857 - type: nauc_recall_at_1_std value: 28.0731 - type: nauc_recall_at_1_diff1 value: 30.3693 - type: nauc_recall_at_3_max value: 29.3966 - type: nauc_recall_at_3_std value: 30.4753 - type: nauc_recall_at_3_diff1 value: 11.6804 - type: nauc_recall_at_5_max value: 25.7178 - type: nauc_recall_at_5_std value: 29.204200000000004 - type: nauc_recall_at_5_diff1 value: 9.138499999999999 - type: nauc_recall_at_10_max value: 23.3461 - type: nauc_recall_at_10_std value: 28.4841 - type: nauc_recall_at_10_diff1 value: 5.3777 - type: nauc_recall_at_20_max value: 20.4754 - type: nauc_recall_at_20_std value: 27.158900000000003 - type: nauc_recall_at_20_diff1 value: 3.728 - type: nauc_recall_at_100_max value: 10.8536 - type: nauc_recall_at_100_std value: 23.548099999999998 - type: nauc_recall_at_100_diff1 value: -2.2279 - type: nauc_recall_at_1000_max value: 3.3938 - type: nauc_recall_at_1000_std value: 18.7659 - type: nauc_recall_at_1000_diff1 value: -11.1298 - type: nauc_precision_at_1_max value: 37.7857 - type: nauc_precision_at_1_std value: 28.0731 - type: nauc_precision_at_1_diff1 value: 30.3693 - type: nauc_precision_at_3_max value: 29.3966 - type: nauc_precision_at_3_std value: 30.4753 - type: nauc_precision_at_3_diff1 value: 11.6804 - type: nauc_precision_at_5_max value: 25.7178 - type: nauc_precision_at_5_std value: 29.204200000000004 - type: nauc_precision_at_5_diff1 value: 9.138499999999999 - type: nauc_precision_at_10_max value: 23.3461 - type: nauc_precision_at_10_std value: 28.4841 - type: nauc_precision_at_10_diff1 value: 5.3777 - type: nauc_precision_at_20_max value: 20.4754 - type: nauc_precision_at_20_std value: 27.158900000000003 - type: nauc_precision_at_20_diff1 value: 3.728 - type: nauc_precision_at_100_max value: 10.8536 - type: nauc_precision_at_100_std value: 23.548099999999998 - type: nauc_precision_at_100_diff1 value: -2.2279 - type: nauc_precision_at_1000_max value: 3.3938 - type: nauc_precision_at_1000_std value: 18.7659 - type: nauc_precision_at_1000_diff1 value: -11.1298 - type: nauc_mrr_at_1_max value: 37.7857 - type: nauc_mrr_at_1_std value: 28.0731 - type: nauc_mrr_at_1_diff1 value: 30.3693 - type: nauc_mrr_at_3_max value: 32.8434 - type: nauc_mrr_at_3_std value: 29.6022 - type: nauc_mrr_at_3_diff1 value: 19.8538 - type: nauc_mrr_at_5_max value: 31.306099999999997 - type: nauc_mrr_at_5_std value: 29.305500000000002 - type: nauc_mrr_at_5_diff1 value: 18.3086 - type: nauc_mrr_at_10_max value: 30.3611 - type: nauc_mrr_at_10_std value: 29.0105 - type: nauc_mrr_at_10_diff1 value: 16.7745 - type: nauc_mrr_at_20_max value: 29.5693 - type: nauc_mrr_at_20_std value: 28.774 - type: nauc_mrr_at_20_diff1 value: 16.084899999999998 - type: nauc_mrr_at_100_max value: 28.548099999999998 - type: nauc_mrr_at_100_std value: 28.522 - type: nauc_mrr_at_100_diff1 value: 15.317900000000002 - type: nauc_mrr_at_1000_max value: 28.412399999999998 - type: nauc_mrr_at_1000_std value: 28.3895 - type: nauc_mrr_at_1000_diff1 value: 15.209600000000002 - type: main_score value: 5.5169999999999995 task: type: Retrieval - dataset: config: ara-hin name: MTEB MLQARetrieval (ara-hin) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 0.7100000000000001 - type: ndcg_at_3 value: 0.979 - type: ndcg_at_5 value: 1.115 - type: ndcg_at_10 value: 1.234 - type: ndcg_at_20 value: 1.477 - type: ndcg_at_100 value: 2.6149999999999998 - type: ndcg_at_1000 value: 9.341000000000001 - type: map_at_1 value: 0.7100000000000001 - type: map_at_3 value: 0.919 - type: map_at_5 value: 0.996 - type: map_at_10 value: 1.042 - type: map_at_20 value: 1.106 - type: map_at_100 value: 1.2349999999999999 - type: map_at_1000 value: 1.391 - type: recall_at_1 value: 0.7100000000000001 - type: recall_at_3 value: 1.147 - type: recall_at_5 value: 1.4749999999999999 - type: recall_at_10 value: 1.857 - type: recall_at_20 value: 2.8400000000000003 - type: recall_at_100 value: 9.394 - type: recall_at_1000 value: 68.214 - type: precision_at_1 value: 0.7100000000000001 - type: precision_at_3 value: 0.382 - type: precision_at_5 value: 0.295 - type: precision_at_10 value: 0.186 - type: precision_at_20 value: 0.14200000000000002 - type: precision_at_100 value: 0.094 - type: precision_at_1000 value: 0.068 - type: mrr_at_1 value: 0.7100000000000001 - type: mrr_at_3 value: 0.9194000000000001 - type: mrr_at_5 value: 0.9958 - type: mrr_at_10 value: 1.0419 - type: mrr_at_20 value: 1.1060999999999999 - type: mrr_at_100 value: 1.2349999999999999 - type: mrr_at_1000 value: 1.3914 - type: nauc_ndcg_at_1_max value: 82.92429999999999 - type: nauc_ndcg_at_1_std value: 87.8644 - type: nauc_ndcg_at_1_diff1 value: 50.809599999999996 - type: nauc_ndcg_at_3_max value: 85.7232 - type: nauc_ndcg_at_3_std value: 77.1186 - type: nauc_ndcg_at_3_diff1 value: 52.585800000000006 - type: nauc_ndcg_at_5_max value: 81.8299 - type: nauc_ndcg_at_5_std value: 75.6657 - type: nauc_ndcg_at_5_diff1 value: 47.604600000000005 - type: nauc_ndcg_at_10_max value: 79.0778 - type: nauc_ndcg_at_10_std value: 73.9233 - type: nauc_ndcg_at_10_diff1 value: 40.9872 - type: nauc_ndcg_at_20_max value: 68.9868 - type: nauc_ndcg_at_20_std value: 62.644999999999996 - type: nauc_ndcg_at_20_diff1 value: 32.3618 - type: nauc_ndcg_at_100_max value: 43.6081 - type: nauc_ndcg_at_100_std value: 41.0802 - type: nauc_ndcg_at_100_diff1 value: 16.6409 - type: nauc_ndcg_at_1000_max value: 29.611500000000003 - type: nauc_ndcg_at_1000_std value: 28.618700000000004 - type: nauc_ndcg_at_1000_diff1 value: 10.985 - type: nauc_map_at_1_max value: 82.92429999999999 - type: nauc_map_at_1_std value: 87.8644 - type: nauc_map_at_1_diff1 value: 50.809599999999996 - type: nauc_map_at_3_max value: 85.474 - type: nauc_map_at_3_std value: 79.1553 - type: nauc_map_at_3_diff1 value: 52.4608 - type: nauc_map_at_5_max value: 83.1901 - type: nauc_map_at_5_std value: 78.1601 - type: nauc_map_at_5_diff1 value: 49.398599999999995 - type: nauc_map_at_10_max value: 81.96679999999999 - type: nauc_map_at_10_std value: 77.3412 - type: nauc_map_at_10_diff1 value: 46.2706 - type: nauc_map_at_20_max value: 78.31240000000001 - type: nauc_map_at_20_std value: 73.1451 - type: nauc_map_at_20_diff1 value: 42.8328 - type: nauc_map_at_100_max value: 71.6632 - type: nauc_map_at_100_std value: 67.2004 - type: nauc_map_at_100_diff1 value: 37.9505 - type: nauc_map_at_1000_max value: 68.9277 - type: nauc_map_at_1000_std value: 64.8685 - type: nauc_map_at_1000_diff1 value: 36.4747 - type: nauc_recall_at_1_max value: 82.92429999999999 - type: nauc_recall_at_1_std value: 87.8644 - type: nauc_recall_at_1_diff1 value: 50.809599999999996 - type: nauc_recall_at_3_max value: 86.2097 - type: nauc_recall_at_3_std value: 72.3955 - type: nauc_recall_at_3_diff1 value: 52.80669999999999 - type: nauc_recall_at_5_max value: 78.6315 - type: nauc_recall_at_5_std value: 70.6 - type: nauc_recall_at_5_diff1 value: 43.5508 - type: nauc_recall_at_10_max value: 73.09740000000001 - type: nauc_recall_at_10_std value: 67.6382 - type: nauc_recall_at_10_diff1 value: 30.257 - type: nauc_recall_at_20_max value: 53.7884 - type: nauc_recall_at_20_std value: 46.1967 - type: nauc_recall_at_20_diff1 value: 16.1309 - type: nauc_recall_at_100_max value: 22.8102 - type: nauc_recall_at_100_std value: 22.4128 - type: nauc_recall_at_100_diff1 value: 2.2661000000000002 - type: nauc_recall_at_1000_max value: 7.2257 - type: nauc_recall_at_1000_std value: 7.8541 - type: nauc_recall_at_1000_diff1 value: -1.7273 - type: nauc_precision_at_1_max value: 82.92429999999999 - type: nauc_precision_at_1_std value: 87.8644 - type: nauc_precision_at_1_diff1 value: 50.809599999999996 - type: nauc_precision_at_3_max value: 86.2097 - type: nauc_precision_at_3_std value: 72.3955 - type: nauc_precision_at_3_diff1 value: 52.80669999999999 - type: nauc_precision_at_5_max value: 78.6315 - type: nauc_precision_at_5_std value: 70.6 - type: nauc_precision_at_5_diff1 value: 43.5508 - type: nauc_precision_at_10_max value: 73.09740000000001 - type: nauc_precision_at_10_std value: 67.6382 - type: nauc_precision_at_10_diff1 value: 30.257 - type: nauc_precision_at_20_max value: 53.7884 - type: nauc_precision_at_20_std value: 46.1967 - type: nauc_precision_at_20_diff1 value: 16.1309 - type: nauc_precision_at_100_max value: 22.8102 - type: nauc_precision_at_100_std value: 22.4128 - type: nauc_precision_at_100_diff1 value: 2.2661000000000002 - type: nauc_precision_at_1000_max value: 7.2257 - type: nauc_precision_at_1000_std value: 7.8541 - type: nauc_precision_at_1000_diff1 value: -1.7273 - type: nauc_mrr_at_1_max value: 82.92429999999999 - type: nauc_mrr_at_1_std value: 87.8644 - type: nauc_mrr_at_1_diff1 value: 50.809599999999996 - type: nauc_mrr_at_3_max value: 85.474 - type: nauc_mrr_at_3_std value: 79.1553 - type: nauc_mrr_at_3_diff1 value: 52.4608 - type: nauc_mrr_at_5_max value: 83.1901 - type: nauc_mrr_at_5_std value: 78.1601 - type: nauc_mrr_at_5_diff1 value: 49.398599999999995 - type: nauc_mrr_at_10_max value: 81.96679999999999 - type: nauc_mrr_at_10_std value: 77.3412 - type: nauc_mrr_at_10_diff1 value: 46.2706 - type: nauc_mrr_at_20_max value: 78.31240000000001 - type: nauc_mrr_at_20_std value: 73.1451 - type: nauc_mrr_at_20_diff1 value: 42.8328 - type: nauc_mrr_at_100_max value: 71.6632 - type: nauc_mrr_at_100_std value: 67.2004 - type: nauc_mrr_at_100_diff1 value: 37.9505 - type: nauc_mrr_at_1000_max value: 68.9277 - type: nauc_mrr_at_1000_std value: 64.8685 - type: nauc_mrr_at_1000_diff1 value: 36.4747 - type: main_score value: 1.234 task: type: Retrieval - dataset: config: ara-vie name: MTEB MLQARetrieval (ara-vie) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 3.517 - type: ndcg_at_3 value: 4.869 - type: ndcg_at_5 value: 5.346 - type: ndcg_at_10 value: 6.151 - type: ndcg_at_20 value: 6.905 - type: ndcg_at_100 value: 8.790000000000001 - type: ndcg_at_1000 value: 14.077 - type: map_at_1 value: 3.517 - type: map_at_3 value: 4.527 - type: map_at_5 value: 4.788 - type: map_at_10 value: 5.12 - type: map_at_20 value: 5.327 - type: map_at_100 value: 5.556 - type: map_at_1000 value: 5.691 - type: recall_at_1 value: 3.517 - type: recall_at_3 value: 5.862 - type: recall_at_5 value: 7.034999999999999 - type: recall_at_10 value: 9.526 - type: recall_at_20 value: 12.506 - type: recall_at_100 value: 23.156 - type: recall_at_1000 value: 68.58800000000001 - type: precision_at_1 value: 3.517 - type: precision_at_3 value: 1.9539999999999997 - type: precision_at_5 value: 1.407 - type: precision_at_10 value: 0.9530000000000001 - type: precision_at_20 value: 0.625 - type: precision_at_100 value: 0.232 - type: precision_at_1000 value: 0.06899999999999999 - type: mrr_at_1 value: 3.5173 - type: mrr_at_3 value: 4.527 - type: mrr_at_5 value: 4.7883000000000004 - type: mrr_at_10 value: 5.12 - type: mrr_at_20 value: 5.3272 - type: mrr_at_100 value: 5.5557 - type: mrr_at_1000 value: 5.6914 - type: nauc_ndcg_at_1_max value: 52.937400000000004 - type: nauc_ndcg_at_1_std value: 10.5306 - type: nauc_ndcg_at_1_diff1 value: 49.3442 - type: nauc_ndcg_at_3_max value: 48.1461 - type: nauc_ndcg_at_3_std value: 14.555299999999999 - type: nauc_ndcg_at_3_diff1 value: 36.0715 - type: nauc_ndcg_at_5_max value: 46.8172 - type: nauc_ndcg_at_5_std value: 15.865000000000002 - type: nauc_ndcg_at_5_diff1 value: 33.0289 - type: nauc_ndcg_at_10_max value: 44.3798 - type: nauc_ndcg_at_10_std value: 17.0958 - type: nauc_ndcg_at_10_diff1 value: 29.937200000000004 - type: nauc_ndcg_at_20_max value: 43.8549 - type: nauc_ndcg_at_20_std value: 18.9531 - type: nauc_ndcg_at_20_diff1 value: 27.3668 - type: nauc_ndcg_at_100_max value: 42.8076 - type: nauc_ndcg_at_100_std value: 21.3048 - type: nauc_ndcg_at_100_diff1 value: 23.416999999999998 - type: nauc_ndcg_at_1000_max value: 41.3446 - type: nauc_ndcg_at_1000_std value: 19.5054 - type: nauc_ndcg_at_1000_diff1 value: 23.210900000000002 - type: nauc_map_at_1_max value: 52.937400000000004 - type: nauc_map_at_1_std value: 10.5306 - type: nauc_map_at_1_diff1 value: 49.3442 - type: nauc_map_at_3_max value: 49.125600000000006 - type: nauc_map_at_3_std value: 13.8065 - type: nauc_map_at_3_diff1 value: 38.649699999999996 - type: nauc_map_at_5_max value: 48.2732 - type: nauc_map_at_5_std value: 14.645 - type: nauc_map_at_5_diff1 value: 36.6398 - type: nauc_map_at_10_max value: 47.0321 - type: nauc_map_at_10_std value: 15.393699999999999 - type: nauc_map_at_10_diff1 value: 34.9069 - type: nauc_map_at_20_max value: 46.7963 - type: nauc_map_at_20_std value: 16.118 - type: nauc_map_at_20_diff1 value: 33.7982 - type: nauc_map_at_100_max value: 46.5471 - type: nauc_map_at_100_std value: 16.612299999999998 - type: nauc_map_at_100_diff1 value: 32.9562 - type: nauc_map_at_1000_max value: 46.456599999999995 - type: nauc_map_at_1000_std value: 16.581799999999998 - type: nauc_map_at_1000_diff1 value: 32.8544 - type: nauc_recall_at_1_max value: 52.937400000000004 - type: nauc_recall_at_1_std value: 10.5306 - type: nauc_recall_at_1_diff1 value: 49.3442 - type: nauc_recall_at_3_max value: 45.9372 - type: nauc_recall_at_3_std value: 16.2224 - type: nauc_recall_at_3_diff1 value: 30.297200000000004 - type: nauc_recall_at_5_max value: 43.7847 - type: nauc_recall_at_5_std value: 18.3997 - type: nauc_recall_at_5_diff1 value: 25.6008 - type: nauc_recall_at_10_max value: 39.5381 - type: nauc_recall_at_10_std value: 20.0222 - type: nauc_recall_at_10_diff1 value: 21.2714 - type: nauc_recall_at_20_max value: 39.157399999999996 - type: nauc_recall_at_20_std value: 23.6117 - type: nauc_recall_at_20_diff1 value: 17.2605 - type: nauc_recall_at_100_max value: 38.115300000000005 - type: nauc_recall_at_100_std value: 27.7982 - type: nauc_recall_at_100_diff1 value: 10.8742 - type: nauc_recall_at_1000_max value: 31.5702 - type: nauc_recall_at_1000_std value: 20.0384 - type: nauc_recall_at_1000_diff1 value: 9.9033 - type: nauc_precision_at_1_max value: 52.937400000000004 - type: nauc_precision_at_1_std value: 10.5306 - type: nauc_precision_at_1_diff1 value: 49.3442 - type: nauc_precision_at_3_max value: 45.9372 - type: nauc_precision_at_3_std value: 16.2224 - type: nauc_precision_at_3_diff1 value: 30.297200000000004 - type: nauc_precision_at_5_max value: 43.7847 - type: nauc_precision_at_5_std value: 18.3997 - type: nauc_precision_at_5_diff1 value: 25.6008 - type: nauc_precision_at_10_max value: 39.5381 - type: nauc_precision_at_10_std value: 20.0222 - type: nauc_precision_at_10_diff1 value: 21.2714 - type: nauc_precision_at_20_max value: 39.157399999999996 - type: nauc_precision_at_20_std value: 23.6117 - type: nauc_precision_at_20_diff1 value: 17.2605 - type: nauc_precision_at_100_max value: 38.115300000000005 - type: nauc_precision_at_100_std value: 27.7982 - type: nauc_precision_at_100_diff1 value: 10.8742 - type: nauc_precision_at_1000_max value: 31.5702 - type: nauc_precision_at_1000_std value: 20.0384 - type: nauc_precision_at_1000_diff1 value: 9.9033 - type: nauc_mrr_at_1_max value: 52.937400000000004 - type: nauc_mrr_at_1_std value: 10.5306 - type: nauc_mrr_at_1_diff1 value: 49.3442 - type: nauc_mrr_at_3_max value: 49.125600000000006 - type: nauc_mrr_at_3_std value: 13.8065 - type: nauc_mrr_at_3_diff1 value: 38.649699999999996 - type: nauc_mrr_at_5_max value: 48.2732 - type: nauc_mrr_at_5_std value: 14.645 - type: nauc_mrr_at_5_diff1 value: 36.6398 - type: nauc_mrr_at_10_max value: 47.0321 - type: nauc_mrr_at_10_std value: 15.393699999999999 - type: nauc_mrr_at_10_diff1 value: 34.9069 - type: nauc_mrr_at_20_max value: 46.7963 - type: nauc_mrr_at_20_std value: 16.118 - type: nauc_mrr_at_20_diff1 value: 33.7982 - type: nauc_mrr_at_100_max value: 46.5471 - type: nauc_mrr_at_100_std value: 16.612299999999998 - type: nauc_mrr_at_100_diff1 value: 32.9562 - type: nauc_mrr_at_1000_max value: 46.456599999999995 - type: nauc_mrr_at_1000_std value: 16.581799999999998 - type: nauc_mrr_at_1000_diff1 value: 32.8544 - type: main_score value: 6.151 task: type: Retrieval - dataset: config: ara-zho name: MTEB MLQARetrieval (ara-zho) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 1.2550000000000001 - type: ndcg_at_3 value: 1.584 - type: ndcg_at_5 value: 1.7319999999999998 - type: ndcg_at_10 value: 1.9300000000000002 - type: ndcg_at_20 value: 2.141 - type: ndcg_at_100 value: 3.2070000000000003 - type: ndcg_at_1000 value: 9.178 - type: map_at_1 value: 1.2550000000000001 - type: map_at_3 value: 1.4989999999999999 - type: map_at_5 value: 1.58 - type: map_at_10 value: 1.659 - type: map_at_20 value: 1.7160000000000002 - type: map_at_100 value: 1.846 - type: map_at_1000 value: 1.9879999999999998 - type: recall_at_1 value: 1.2550000000000001 - type: recall_at_3 value: 1.831 - type: recall_at_5 value: 2.197 - type: recall_at_10 value: 2.8240000000000003 - type: recall_at_20 value: 3.6609999999999996 - type: recall_at_100 value: 9.676 - type: recall_at_1000 value: 61.82 - type: precision_at_1 value: 1.2550000000000001 - type: precision_at_3 value: 0.61 - type: precision_at_5 value: 0.439 - type: precision_at_10 value: 0.28200000000000003 - type: precision_at_20 value: 0.183 - type: precision_at_100 value: 0.097 - type: precision_at_1000 value: 0.062 - type: mrr_at_1 value: 1.2552 - type: mrr_at_3 value: 1.4992999999999999 - type: mrr_at_5 value: 1.5803999999999998 - type: mrr_at_10 value: 1.6587999999999998 - type: mrr_at_20 value: 1.7159 - type: mrr_at_100 value: 1.8458999999999999 - type: mrr_at_1000 value: 1.9876999999999998 - type: nauc_ndcg_at_1_max value: 98.5906 - type: nauc_ndcg_at_1_std value: 86.87979999999999 - type: nauc_ndcg_at_1_diff1 value: 80.9805 - type: nauc_ndcg_at_3_max value: 93.8088 - type: nauc_ndcg_at_3_std value: 83.8238 - type: nauc_ndcg_at_3_diff1 value: 73.03450000000001 - type: nauc_ndcg_at_5_max value: 91.522 - type: nauc_ndcg_at_5_std value: 78.068 - type: nauc_ndcg_at_5_diff1 value: 69.3089 - type: nauc_ndcg_at_10_max value: 87.6146 - type: nauc_ndcg_at_10_std value: 72.7358 - type: nauc_ndcg_at_10_diff1 value: 66.0461 - type: nauc_ndcg_at_20_max value: 83.10419999999999 - type: nauc_ndcg_at_20_std value: 68.87180000000001 - type: nauc_ndcg_at_20_diff1 value: 61.0792 - type: nauc_ndcg_at_100_max value: 63.6503 - type: nauc_ndcg_at_100_std value: 53.172799999999995 - type: nauc_ndcg_at_100_diff1 value: 44.6957 - type: nauc_ndcg_at_1000_max value: 44.885799999999996 - type: nauc_ndcg_at_1000_std value: 38.0912 - type: nauc_ndcg_at_1000_diff1 value: 31.901600000000002 - type: nauc_map_at_1_max value: 98.5906 - type: nauc_map_at_1_std value: 86.87979999999999 - type: nauc_map_at_1_diff1 value: 80.9805 - type: nauc_map_at_3_max value: 95.0588 - type: nauc_map_at_3_std value: 84.6644 - type: nauc_map_at_3_diff1 value: 75.0602 - type: nauc_map_at_5_max value: 93.6339 - type: nauc_map_at_5_std value: 81.2838 - type: nauc_map_at_5_diff1 value: 72.6508 - type: nauc_map_at_10_max value: 91.75710000000001 - type: nauc_map_at_10_std value: 78.6057 - type: nauc_map_at_10_diff1 value: 70.9955 - type: nauc_map_at_20_max value: 89.992 - type: nauc_map_at_20_std value: 76.99780000000001 - type: nauc_map_at_20_diff1 value: 69.1547 - type: nauc_map_at_100_max value: 85.8335 - type: nauc_map_at_100_std value: 73.555 - type: nauc_map_at_100_diff1 value: 65.4493 - type: nauc_map_at_1000_max value: 83.6392 - type: nauc_map_at_1000_std value: 71.6168 - type: nauc_map_at_1000_diff1 value: 63.7131 - type: nauc_recall_at_1_max value: 98.5906 - type: nauc_recall_at_1_std value: 86.87979999999999 - type: nauc_recall_at_1_diff1 value: 80.9805 - type: nauc_recall_at_3_max value: 90.7581 - type: nauc_recall_at_3_std value: 81.7614 - type: nauc_recall_at_3_diff1 value: 68.1038 - type: nauc_recall_at_5_max value: 86.7975 - type: nauc_recall_at_5_std value: 70.5951 - type: nauc_recall_at_5_diff1 value: 61.908300000000004 - type: nauc_recall_at_10_max value: 79.29079999999999 - type: nauc_recall_at_10_std value: 60.8799 - type: nauc_recall_at_10_diff1 value: 56.4518 - type: nauc_recall_at_20_max value: 71.0629 - type: nauc_recall_at_20_std value: 54.98610000000001 - type: nauc_recall_at_20_diff1 value: 47.058299999999996 - type: nauc_recall_at_100_max value: 40.8869 - type: nauc_recall_at_100_std value: 32.9852 - type: nauc_recall_at_100_diff1 value: 24.1605 - type: nauc_recall_at_1000_max value: 15.2075 - type: nauc_recall_at_1000_std value: 13.9598 - type: nauc_recall_at_1000_diff1 value: 9.5109 - type: nauc_precision_at_1_max value: 98.5906 - type: nauc_precision_at_1_std value: 86.87979999999999 - type: nauc_precision_at_1_diff1 value: 80.9805 - type: nauc_precision_at_3_max value: 90.7581 - type: nauc_precision_at_3_std value: 81.7614 - type: nauc_precision_at_3_diff1 value: 68.1038 - type: nauc_precision_at_5_max value: 86.7975 - type: nauc_precision_at_5_std value: 70.5951 - type: nauc_precision_at_5_diff1 value: 61.908300000000004 - type: nauc_precision_at_10_max value: 79.29079999999999 - type: nauc_precision_at_10_std value: 60.8799 - type: nauc_precision_at_10_diff1 value: 56.4518 - type: nauc_precision_at_20_max value: 71.0629 - type: nauc_precision_at_20_std value: 54.98610000000001 - type: nauc_precision_at_20_diff1 value: 47.058299999999996 - type: nauc_precision_at_100_max value: 40.8869 - type: nauc_precision_at_100_std value: 32.9852 - type: nauc_precision_at_100_diff1 value: 24.1605 - type: nauc_precision_at_1000_max value: 15.2075 - type: nauc_precision_at_1000_std value: 13.9598 - type: nauc_precision_at_1000_diff1 value: 9.5109 - type: nauc_mrr_at_1_max value: 98.5906 - type: nauc_mrr_at_1_std value: 86.87979999999999 - type: nauc_mrr_at_1_diff1 value: 80.9805 - type: nauc_mrr_at_3_max value: 95.0588 - type: nauc_mrr_at_3_std value: 84.6644 - type: nauc_mrr_at_3_diff1 value: 75.0602 - type: nauc_mrr_at_5_max value: 93.6339 - type: nauc_mrr_at_5_std value: 81.2838 - type: nauc_mrr_at_5_diff1 value: 72.6508 - type: nauc_mrr_at_10_max value: 91.75710000000001 - type: nauc_mrr_at_10_std value: 78.6057 - type: nauc_mrr_at_10_diff1 value: 70.9955 - type: nauc_mrr_at_20_max value: 89.992 - type: nauc_mrr_at_20_std value: 76.99780000000001 - type: nauc_mrr_at_20_diff1 value: 69.1547 - type: nauc_mrr_at_100_max value: 85.8335 - type: nauc_mrr_at_100_std value: 73.555 - type: nauc_mrr_at_100_diff1 value: 65.4493 - type: nauc_mrr_at_1000_max value: 83.6392 - type: nauc_mrr_at_1000_std value: 71.6168 - type: nauc_mrr_at_1000_diff1 value: 63.7131 - type: main_score value: 1.9300000000000002 task: type: Retrieval - dataset: config: deu-ara name: MTEB MLQARetrieval (deu-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 7.8839999999999995 - type: ndcg_at_3 value: 10.482 - type: ndcg_at_5 value: 11.254 - type: ndcg_at_10 value: 12.598 - type: ndcg_at_20 value: 13.988999999999999 - type: ndcg_at_100 value: 17.518 - type: ndcg_at_1000 value: 23.29 - type: map_at_1 value: 7.8839999999999995 - type: map_at_3 value: 9.794 - type: map_at_5 value: 10.221 - type: map_at_10 value: 10.771 - type: map_at_20 value: 11.158 - type: map_at_100 value: 11.608 - type: map_at_1000 value: 11.772 - type: recall_at_1 value: 7.8839999999999995 - type: recall_at_3 value: 12.492 - type: recall_at_5 value: 14.371999999999998 - type: recall_at_10 value: 18.557000000000002 - type: recall_at_20 value: 24.015 - type: recall_at_100 value: 43.602000000000004 - type: recall_at_1000 value: 92.17699999999999 - type: precision_at_1 value: 7.8839999999999995 - type: precision_at_3 value: 4.164000000000001 - type: precision_at_5 value: 2.874 - type: precision_at_10 value: 1.856 - type: precision_at_20 value: 1.201 - type: precision_at_100 value: 0.436 - type: precision_at_1000 value: 0.092 - type: mrr_at_1 value: 7.8836 - type: mrr_at_3 value: 9.7938 - type: mrr_at_5 value: 10.2213 - type: mrr_at_10 value: 10.7712 - type: mrr_at_20 value: 11.1583 - type: mrr_at_100 value: 11.6083 - type: mrr_at_1000 value: 11.772 - type: nauc_ndcg_at_1_max value: 39.684599999999996 - type: nauc_ndcg_at_1_std value: 16.4453 - type: nauc_ndcg_at_1_diff1 value: 41.5161 - type: nauc_ndcg_at_3_max value: 36.9155 - type: nauc_ndcg_at_3_std value: 14.8799 - type: nauc_ndcg_at_3_diff1 value: 34.2639 - type: nauc_ndcg_at_5_max value: 35.7577 - type: nauc_ndcg_at_5_std value: 15.592 - type: nauc_ndcg_at_5_diff1 value: 32.8083 - type: nauc_ndcg_at_10_max value: 34.3496 - type: nauc_ndcg_at_10_std value: 15.763399999999999 - type: nauc_ndcg_at_10_diff1 value: 30.444900000000004 - type: nauc_ndcg_at_20_max value: 33.4882 - type: nauc_ndcg_at_20_std value: 15.7758 - type: nauc_ndcg_at_20_diff1 value: 29.4493 - type: nauc_ndcg_at_100_max value: 33.387699999999995 - type: nauc_ndcg_at_100_std value: 16.9045 - type: nauc_ndcg_at_100_diff1 value: 27.5416 - type: nauc_ndcg_at_1000_max value: 34.324799999999996 - type: nauc_ndcg_at_1000_std value: 16.9937 - type: nauc_ndcg_at_1000_diff1 value: 29.342800000000004 - type: nauc_map_at_1_max value: 39.684599999999996 - type: nauc_map_at_1_std value: 16.4453 - type: nauc_map_at_1_diff1 value: 41.5161 - type: nauc_map_at_3_max value: 37.3834 - type: nauc_map_at_3_std value: 15.137400000000001 - type: nauc_map_at_3_diff1 value: 35.694900000000004 - type: nauc_map_at_5_max value: 36.6924 - type: nauc_map_at_5_std value: 15.540499999999998 - type: nauc_map_at_5_diff1 value: 34.821000000000005 - type: nauc_map_at_10_max value: 35.9902 - type: nauc_map_at_10_std value: 15.590200000000001 - type: nauc_map_at_10_diff1 value: 33.6344 - type: nauc_map_at_20_max value: 35.6531 - type: nauc_map_at_20_std value: 15.593399999999999 - type: nauc_map_at_20_diff1 value: 33.2171 - type: nauc_map_at_100_max value: 35.5569 - type: nauc_map_at_100_std value: 15.6981 - type: nauc_map_at_100_diff1 value: 32.8419 - type: nauc_map_at_1000_max value: 35.596 - type: nauc_map_at_1000_std value: 15.712699999999998 - type: nauc_map_at_1000_diff1 value: 32.8887 - type: nauc_recall_at_1_max value: 39.684599999999996 - type: nauc_recall_at_1_std value: 16.4453 - type: nauc_recall_at_1_diff1 value: 41.5161 - type: nauc_recall_at_3_max value: 35.842800000000004 - type: nauc_recall_at_3_std value: 14.292399999999999 - type: nauc_recall_at_3_diff1 value: 30.9013 - type: nauc_recall_at_5_max value: 33.5924 - type: nauc_recall_at_5_std value: 15.815999999999999 - type: nauc_recall_at_5_diff1 value: 28.2062 - type: nauc_recall_at_10_max value: 30.8288 - type: nauc_recall_at_10_std value: 16.272100000000002 - type: nauc_recall_at_10_diff1 value: 23.7129 - type: nauc_recall_at_20_max value: 29.1273 - type: nauc_recall_at_20_std value: 16.2601 - type: nauc_recall_at_20_diff1 value: 22.1143 - type: nauc_recall_at_100_max value: 29.833 - type: nauc_recall_at_100_std value: 20.6089 - type: nauc_recall_at_100_diff1 value: 16.9861 - type: nauc_recall_at_1000_max value: 34.473 - type: nauc_recall_at_1000_std value: 32.7194 - type: nauc_recall_at_1000_diff1 value: 18.7253 - type: nauc_precision_at_1_max value: 39.684599999999996 - type: nauc_precision_at_1_std value: 16.4453 - type: nauc_precision_at_1_diff1 value: 41.5161 - type: nauc_precision_at_3_max value: 35.842800000000004 - type: nauc_precision_at_3_std value: 14.292399999999999 - type: nauc_precision_at_3_diff1 value: 30.9013 - type: nauc_precision_at_5_max value: 33.5924 - type: nauc_precision_at_5_std value: 15.815999999999999 - type: nauc_precision_at_5_diff1 value: 28.2062 - type: nauc_precision_at_10_max value: 30.8288 - type: nauc_precision_at_10_std value: 16.272100000000002 - type: nauc_precision_at_10_diff1 value: 23.7129 - type: nauc_precision_at_20_max value: 29.1273 - type: nauc_precision_at_20_std value: 16.2601 - type: nauc_precision_at_20_diff1 value: 22.1143 - type: nauc_precision_at_100_max value: 29.833 - type: nauc_precision_at_100_std value: 20.6089 - type: nauc_precision_at_100_diff1 value: 16.9861 - type: nauc_precision_at_1000_max value: 34.473 - type: nauc_precision_at_1000_std value: 32.7194 - type: nauc_precision_at_1000_diff1 value: 18.7253 - type: nauc_mrr_at_1_max value: 39.684599999999996 - type: nauc_mrr_at_1_std value: 16.4453 - type: nauc_mrr_at_1_diff1 value: 41.5161 - type: nauc_mrr_at_3_max value: 37.3834 - type: nauc_mrr_at_3_std value: 15.137400000000001 - type: nauc_mrr_at_3_diff1 value: 35.694900000000004 - type: nauc_mrr_at_5_max value: 36.6924 - type: nauc_mrr_at_5_std value: 15.540499999999998 - type: nauc_mrr_at_5_diff1 value: 34.821000000000005 - type: nauc_mrr_at_10_max value: 35.9902 - type: nauc_mrr_at_10_std value: 15.590200000000001 - type: nauc_mrr_at_10_diff1 value: 33.6344 - type: nauc_mrr_at_20_max value: 35.6531 - type: nauc_mrr_at_20_std value: 15.593399999999999 - type: nauc_mrr_at_20_diff1 value: 33.2171 - type: nauc_mrr_at_100_max value: 35.5569 - type: nauc_mrr_at_100_std value: 15.6981 - type: nauc_mrr_at_100_diff1 value: 32.8419 - type: nauc_mrr_at_1000_max value: 35.596 - type: nauc_mrr_at_1000_std value: 15.712699999999998 - type: nauc_mrr_at_1000_diff1 value: 32.8887 - type: main_score value: 12.598 task: type: Retrieval - dataset: config: eng-ara name: MTEB MLQARetrieval (eng-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 10.069 - type: ndcg_at_3 value: 14.071 - type: ndcg_at_5 value: 15.692999999999998 - type: ndcg_at_10 value: 17.519000000000002 - type: ndcg_at_20 value: 19.076999999999998 - type: ndcg_at_100 value: 22.213 - type: ndcg_at_1000 value: 26.237 - type: map_at_1 value: 10.069 - type: map_at_3 value: 13.056000000000001 - type: map_at_5 value: 13.950999999999999 - type: map_at_10 value: 14.706 - type: map_at_20 value: 15.132000000000001 - type: map_at_100 value: 15.551 - type: map_at_1000 value: 15.677 - type: recall_at_1 value: 10.069 - type: recall_at_3 value: 17.017 - type: recall_at_5 value: 20.973 - type: recall_at_10 value: 26.617 - type: recall_at_20 value: 32.786 - type: recall_at_100 value: 49.916 - type: recall_at_1000 value: 83.068 - type: precision_at_1 value: 10.069 - type: precision_at_3 value: 5.675 - type: precision_at_5 value: 4.197 - type: precision_at_10 value: 2.6630000000000003 - type: precision_at_20 value: 1.6400000000000001 - type: precision_at_100 value: 0.49899999999999994 - type: precision_at_1000 value: 0.083 - type: mrr_at_1 value: 10.0694 - type: mrr_at_3 value: 13.0602 - type: mrr_at_5 value: 13.9556 - type: mrr_at_10 value: 14.710500000000001 - type: mrr_at_20 value: 15.1367 - type: mrr_at_100 value: 15.5551 - type: mrr_at_1000 value: 15.6807 - type: nauc_ndcg_at_1_max value: 38.708 - type: nauc_ndcg_at_1_std value: 19.6744 - type: nauc_ndcg_at_1_diff1 value: 43.8754 - type: nauc_ndcg_at_3_max value: 35.5438 - type: nauc_ndcg_at_3_std value: 20.7335 - type: nauc_ndcg_at_3_diff1 value: 34.7409 - type: nauc_ndcg_at_5_max value: 33.9112 - type: nauc_ndcg_at_5_std value: 20.8953 - type: nauc_ndcg_at_5_diff1 value: 32.7973 - type: nauc_ndcg_at_10_max value: 33.4574 - type: nauc_ndcg_at_10_std value: 21.407999999999998 - type: nauc_ndcg_at_10_diff1 value: 31.5918 - type: nauc_ndcg_at_20_max value: 33.118900000000004 - type: nauc_ndcg_at_20_std value: 22.0212 - type: nauc_ndcg_at_20_diff1 value: 30.415999999999997 - type: nauc_ndcg_at_100_max value: 32.3823 - type: nauc_ndcg_at_100_std value: 23.1147 - type: nauc_ndcg_at_100_diff1 value: 29.250300000000003 - type: nauc_ndcg_at_1000_max value: 33.2213 - type: nauc_ndcg_at_1000_std value: 23.461199999999998 - type: nauc_ndcg_at_1000_diff1 value: 30.375400000000003 - type: nauc_map_at_1_max value: 38.708 - type: nauc_map_at_1_std value: 19.6744 - type: nauc_map_at_1_diff1 value: 43.8754 - type: nauc_map_at_3_max value: 36.2717 - type: nauc_map_at_3_std value: 20.555300000000003 - type: nauc_map_at_3_diff1 value: 36.670700000000004 - type: nauc_map_at_5_max value: 35.2528 - type: nauc_map_at_5_std value: 20.673 - type: nauc_map_at_5_diff1 value: 35.4282 - type: nauc_map_at_10_max value: 35.0175 - type: nauc_map_at_10_std value: 20.9185 - type: nauc_map_at_10_diff1 value: 34.8249 - type: nauc_map_at_20_max value: 34.901199999999996 - type: nauc_map_at_20_std value: 21.1207 - type: nauc_map_at_20_diff1 value: 34.430899999999994 - type: nauc_map_at_100_max value: 34.7715 - type: nauc_map_at_100_std value: 21.2502 - type: nauc_map_at_100_diff1 value: 34.2391 - type: nauc_map_at_1000_max value: 34.806799999999996 - type: nauc_map_at_1000_std value: 21.2741 - type: nauc_map_at_1000_diff1 value: 34.2916 - type: nauc_recall_at_1_max value: 38.708 - type: nauc_recall_at_1_std value: 19.6744 - type: nauc_recall_at_1_diff1 value: 43.8754 - type: nauc_recall_at_3_max value: 33.805600000000005 - type: nauc_recall_at_3_std value: 21.1238 - type: nauc_recall_at_3_diff1 value: 30.1668 - type: nauc_recall_at_5_max value: 30.8531 - type: nauc_recall_at_5_std value: 21.3447 - type: nauc_recall_at_5_diff1 value: 26.9456 - type: nauc_recall_at_10_max value: 30.0917 - type: nauc_recall_at_10_std value: 22.483900000000002 - type: nauc_recall_at_10_diff1 value: 24.6844 - type: nauc_recall_at_20_max value: 29.3467 - type: nauc_recall_at_20_std value: 24.1344 - type: nauc_recall_at_20_diff1 value: 21.7971 - type: nauc_recall_at_100_max value: 26.6242 - type: nauc_recall_at_100_std value: 28.8262 - type: nauc_recall_at_100_diff1 value: 17.282600000000002 - type: nauc_recall_at_1000_max value: 29.540399999999998 - type: nauc_recall_at_1000_std value: 38.1484 - type: nauc_recall_at_1000_diff1 value: 16.1682 - type: nauc_precision_at_1_max value: 38.708 - type: nauc_precision_at_1_std value: 19.6744 - type: nauc_precision_at_1_diff1 value: 43.8754 - type: nauc_precision_at_3_max value: 33.7781 - type: nauc_precision_at_3_std value: 21.1572 - type: nauc_precision_at_3_diff1 value: 30.122 - type: nauc_precision_at_5_max value: 30.8362 - type: nauc_precision_at_5_std value: 21.3776 - type: nauc_precision_at_5_diff1 value: 26.9131 - type: nauc_precision_at_10_max value: 30.0766 - type: nauc_precision_at_10_std value: 22.5133 - type: nauc_precision_at_10_diff1 value: 24.6554 - type: nauc_precision_at_20_max value: 29.3358 - type: nauc_precision_at_20_std value: 24.164099999999998 - type: nauc_precision_at_20_diff1 value: 21.7729 - type: nauc_precision_at_100_max value: 26.5803 - type: nauc_precision_at_100_std value: 28.8609 - type: nauc_precision_at_100_diff1 value: 17.2201 - type: nauc_precision_at_1000_max value: 29.444599999999998 - type: nauc_precision_at_1000_std value: 38.1935 - type: nauc_precision_at_1000_diff1 value: 16.067 - type: nauc_mrr_at_1_max value: 38.708 - type: nauc_mrr_at_1_std value: 19.6744 - type: nauc_mrr_at_1_diff1 value: 43.8754 - type: nauc_mrr_at_3_max value: 36.2544 - type: nauc_mrr_at_3_std value: 20.5758 - type: nauc_mrr_at_3_diff1 value: 36.6418 - type: nauc_mrr_at_5_max value: 35.2364 - type: nauc_mrr_at_5_std value: 20.6926 - type: nauc_mrr_at_5_diff1 value: 35.4008 - type: nauc_mrr_at_10_max value: 35.0015 - type: nauc_mrr_at_10_std value: 20.9377 - type: nauc_mrr_at_10_diff1 value: 34.7981 - type: nauc_mrr_at_20_max value: 34.8853 - type: nauc_mrr_at_20_std value: 21.1397 - type: nauc_mrr_at_20_diff1 value: 34.4044 - type: nauc_mrr_at_100_max value: 34.7576 - type: nauc_mrr_at_100_std value: 21.2668 - type: nauc_mrr_at_100_diff1 value: 34.216 - type: nauc_mrr_at_1000_max value: 34.792899999999996 - type: nauc_mrr_at_1000_std value: 21.290899999999997 - type: nauc_mrr_at_1000_diff1 value: 34.2684 - type: main_score value: 17.519000000000002 task: type: Retrieval - dataset: config: spa-ara name: MTEB MLQARetrieval (spa-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 5.106 - type: ndcg_at_3 value: 7.369000000000001 - type: ndcg_at_5 value: 8.505 - type: ndcg_at_10 value: 9.411 - type: ndcg_at_20 value: 10.693999999999999 - type: ndcg_at_100 value: 13.506000000000002 - type: ndcg_at_1000 value: 19.200999999999997 - type: map_at_1 value: 5.106 - type: map_at_3 value: 6.7909999999999995 - type: map_at_5 value: 7.428 - type: map_at_10 value: 7.786999999999999 - type: map_at_20 value: 8.135 - type: map_at_100 value: 8.487 - type: map_at_1000 value: 8.648 - type: recall_at_1 value: 5.106 - type: recall_at_3 value: 9.049999999999999 - type: recall_at_5 value: 11.78 - type: recall_at_10 value: 14.661 - type: recall_at_20 value: 19.767000000000003 - type: recall_at_100 value: 35.49 - type: recall_at_1000 value: 83.519 - type: precision_at_1 value: 5.106 - type: precision_at_3 value: 3.017 - type: precision_at_5 value: 2.3560000000000003 - type: precision_at_10 value: 1.466 - type: precision_at_20 value: 0.988 - type: precision_at_100 value: 0.35500000000000004 - type: precision_at_1000 value: 0.084 - type: mrr_at_1 value: 5.1062 - type: mrr_at_3 value: 6.7914 - type: mrr_at_5 value: 7.4284 - type: mrr_at_10 value: 7.786999999999999 - type: mrr_at_20 value: 8.135100000000001 - type: mrr_at_100 value: 8.486699999999999 - type: mrr_at_1000 value: 8.6481 - type: nauc_ndcg_at_1_max value: 42.0862 - type: nauc_ndcg_at_1_std value: 27.788200000000003 - type: nauc_ndcg_at_1_diff1 value: 34.989599999999996 - type: nauc_ndcg_at_3_max value: 34.0335 - type: nauc_ndcg_at_3_std value: 22.6509 - type: nauc_ndcg_at_3_diff1 value: 24.2536 - type: nauc_ndcg_at_5_max value: 35.6463 - type: nauc_ndcg_at_5_std value: 23.9951 - type: nauc_ndcg_at_5_diff1 value: 23.3768 - type: nauc_ndcg_at_10_max value: 34.856300000000005 - type: nauc_ndcg_at_10_std value: 24.4421 - type: nauc_ndcg_at_10_diff1 value: 21.6709 - type: nauc_ndcg_at_20_max value: 32.737899999999996 - type: nauc_ndcg_at_20_std value: 23.2538 - type: nauc_ndcg_at_20_diff1 value: 20.0912 - type: nauc_ndcg_at_100_max value: 32.892900000000004 - type: nauc_ndcg_at_100_std value: 24.5202 - type: nauc_ndcg_at_100_diff1 value: 18.8143 - type: nauc_ndcg_at_1000_max value: 34.2888 - type: nauc_ndcg_at_1000_std value: 25.563000000000002 - type: nauc_ndcg_at_1000_diff1 value: 19.6998 - type: nauc_map_at_1_max value: 42.0862 - type: nauc_map_at_1_std value: 27.788200000000003 - type: nauc_map_at_1_diff1 value: 34.989599999999996 - type: nauc_map_at_3_max value: 35.6994 - type: nauc_map_at_3_std value: 23.6218 - type: nauc_map_at_3_diff1 value: 26.1723 - type: nauc_map_at_5_max value: 36.6046 - type: nauc_map_at_5_std value: 24.413999999999998 - type: nauc_map_at_5_diff1 value: 25.526300000000003 - type: nauc_map_at_10_max value: 36.174299999999995 - type: nauc_map_at_10_std value: 24.5514 - type: nauc_map_at_10_diff1 value: 24.6667 - type: nauc_map_at_20_max value: 35.4277 - type: nauc_map_at_20_std value: 24.1426 - type: nauc_map_at_20_diff1 value: 24.0337 - type: nauc_map_at_100_max value: 35.4035 - type: nauc_map_at_100_std value: 24.4019 - type: nauc_map_at_100_diff1 value: 23.7328 - type: nauc_map_at_1000_max value: 35.4478 - type: nauc_map_at_1000_std value: 24.4698 - type: nauc_map_at_1000_diff1 value: 23.7338 - type: nauc_recall_at_1_max value: 42.0862 - type: nauc_recall_at_1_std value: 27.788200000000003 - type: nauc_recall_at_1_diff1 value: 34.989599999999996 - type: nauc_recall_at_3_max value: 30.361700000000003 - type: nauc_recall_at_3_std value: 20.535800000000002 - type: nauc_recall_at_3_diff1 value: 20.1071 - type: nauc_recall_at_5_max value: 33.937200000000004 - type: nauc_recall_at_5_std value: 23.3113 - type: nauc_recall_at_5_diff1 value: 19.224 - type: nauc_recall_at_10_max value: 32.582499999999996 - type: nauc_recall_at_10_std value: 24.5474 - type: nauc_recall_at_10_diff1 value: 16.14 - type: nauc_recall_at_20_max value: 28.0364 - type: nauc_recall_at_20_std value: 21.7445 - type: nauc_recall_at_20_diff1 value: 13.5925 - type: nauc_recall_at_100_max value: 29.497899999999998 - type: nauc_recall_at_100_std value: 25.3471 - type: nauc_recall_at_100_diff1 value: 11.39 - type: nauc_recall_at_1000_max value: 37.4425 - type: nauc_recall_at_1000_std value: 33.4301 - type: nauc_recall_at_1000_diff1 value: 11.6744 - type: nauc_precision_at_1_max value: 42.0862 - type: nauc_precision_at_1_std value: 27.788200000000003 - type: nauc_precision_at_1_diff1 value: 34.989599999999996 - type: nauc_precision_at_3_max value: 30.361700000000003 - type: nauc_precision_at_3_std value: 20.535800000000002 - type: nauc_precision_at_3_diff1 value: 20.1071 - type: nauc_precision_at_5_max value: 33.937200000000004 - type: nauc_precision_at_5_std value: 23.3113 - type: nauc_precision_at_5_diff1 value: 19.224 - type: nauc_precision_at_10_max value: 32.582499999999996 - type: nauc_precision_at_10_std value: 24.5474 - type: nauc_precision_at_10_diff1 value: 16.14 - type: nauc_precision_at_20_max value: 28.0364 - type: nauc_precision_at_20_std value: 21.7445 - type: nauc_precision_at_20_diff1 value: 13.5925 - type: nauc_precision_at_100_max value: 29.497899999999998 - type: nauc_precision_at_100_std value: 25.3471 - type: nauc_precision_at_100_diff1 value: 11.39 - type: nauc_precision_at_1000_max value: 37.4425 - type: nauc_precision_at_1000_std value: 33.4301 - type: nauc_precision_at_1000_diff1 value: 11.6744 - type: nauc_mrr_at_1_max value: 42.0862 - type: nauc_mrr_at_1_std value: 27.788200000000003 - type: nauc_mrr_at_1_diff1 value: 34.989599999999996 - type: nauc_mrr_at_3_max value: 35.6994 - type: nauc_mrr_at_3_std value: 23.6218 - type: nauc_mrr_at_3_diff1 value: 26.1723 - type: nauc_mrr_at_5_max value: 36.6046 - type: nauc_mrr_at_5_std value: 24.413999999999998 - type: nauc_mrr_at_5_diff1 value: 25.526300000000003 - type: nauc_mrr_at_10_max value: 36.174299999999995 - type: nauc_mrr_at_10_std value: 24.5514 - type: nauc_mrr_at_10_diff1 value: 24.6667 - type: nauc_mrr_at_20_max value: 35.4277 - type: nauc_mrr_at_20_std value: 24.1426 - type: nauc_mrr_at_20_diff1 value: 24.0337 - type: nauc_mrr_at_100_max value: 35.4035 - type: nauc_mrr_at_100_std value: 24.4019 - type: nauc_mrr_at_100_diff1 value: 23.7328 - type: nauc_mrr_at_1000_max value: 35.4478 - type: nauc_mrr_at_1000_std value: 24.4698 - type: nauc_mrr_at_1000_diff1 value: 23.7338 - type: main_score value: 9.411 task: type: Retrieval - dataset: config: hin-ara name: MTEB MLQARetrieval (hin-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 3.659 - type: ndcg_at_3 value: 4.634 - type: ndcg_at_5 value: 4.987 - type: ndcg_at_10 value: 5.431 - type: ndcg_at_20 value: 6.136 - type: ndcg_at_100 value: 8.109 - type: ndcg_at_1000 value: 14.792 - type: map_at_1 value: 3.659 - type: map_at_3 value: 4.387 - type: map_at_5 value: 4.579 - type: map_at_10 value: 4.753 - type: map_at_20 value: 4.9399999999999995 - type: map_at_100 value: 5.178 - type: map_at_1000 value: 5.35 - type: recall_at_1 value: 3.659 - type: recall_at_3 value: 5.351999999999999 - type: recall_at_5 value: 6.226 - type: recall_at_10 value: 7.646 - type: recall_at_20 value: 10.485999999999999 - type: recall_at_100 value: 21.628 - type: recall_at_1000 value: 79.082 - type: precision_at_1 value: 3.659 - type: precision_at_3 value: 1.7840000000000003 - type: precision_at_5 value: 1.2449999999999999 - type: precision_at_10 value: 0.765 - type: precision_at_20 value: 0.524 - type: precision_at_100 value: 0.216 - type: precision_at_1000 value: 0.079 - type: mrr_at_1 value: 3.6592 - type: mrr_at_3 value: 4.3874 - type: mrr_at_5 value: 4.5786 - type: mrr_at_10 value: 4.7534 - type: mrr_at_20 value: 4.9397 - type: mrr_at_100 value: 5.1784 - type: mrr_at_1000 value: 5.3503 - type: nauc_ndcg_at_1_max value: 44.858 - type: nauc_ndcg_at_1_std value: 12.721499999999999 - type: nauc_ndcg_at_1_diff1 value: 41.9307 - type: nauc_ndcg_at_3_max value: 38.853300000000004 - type: nauc_ndcg_at_3_std value: 13.910900000000002 - type: nauc_ndcg_at_3_diff1 value: 32.8699 - type: nauc_ndcg_at_5_max value: 36.6179 - type: nauc_ndcg_at_5_std value: 12.9273 - type: nauc_ndcg_at_5_diff1 value: 30.491699999999998 - type: nauc_ndcg_at_10_max value: 33.8031 - type: nauc_ndcg_at_10_std value: 11.767900000000001 - type: nauc_ndcg_at_10_diff1 value: 27.305600000000002 - type: nauc_ndcg_at_20_max value: 31.2789 - type: nauc_ndcg_at_20_std value: 12.3283 - type: nauc_ndcg_at_20_diff1 value: 24.241699999999998 - type: nauc_ndcg_at_100_max value: 29.2713 - type: nauc_ndcg_at_100_std value: 15.3666 - type: nauc_ndcg_at_100_diff1 value: 20.3282 - type: nauc_ndcg_at_1000_max value: 28.8277 - type: nauc_ndcg_at_1000_std value: 13.9692 - type: nauc_ndcg_at_1000_diff1 value: 19.7898 - type: nauc_map_at_1_max value: 44.858 - type: nauc_map_at_1_std value: 12.721499999999999 - type: nauc_map_at_1_diff1 value: 41.9307 - type: nauc_map_at_3_max value: 40.1323 - type: nauc_map_at_3_std value: 13.677200000000001 - type: nauc_map_at_3_diff1 value: 34.828399999999995 - type: nauc_map_at_5_max value: 38.7125 - type: nauc_map_at_5_std value: 13.070300000000001 - type: nauc_map_at_5_diff1 value: 33.3574 - type: nauc_map_at_10_max value: 37.3599 - type: nauc_map_at_10_std value: 12.5138 - type: nauc_map_at_10_diff1 value: 31.848599999999998 - type: nauc_map_at_20_max value: 36.3682 - type: nauc_map_at_20_std value: 12.6613 - type: nauc_map_at_20_diff1 value: 30.641000000000002 - type: nauc_map_at_100_max value: 35.8721 - type: nauc_map_at_100_std value: 13.1938 - type: nauc_map_at_100_diff1 value: 29.8344 - type: nauc_map_at_1000_max value: 35.7943 - type: nauc_map_at_1000_std value: 13.2437 - type: nauc_map_at_1000_diff1 value: 29.710199999999997 - type: nauc_recall_at_1_max value: 44.858 - type: nauc_recall_at_1_std value: 12.721499999999999 - type: nauc_recall_at_1_diff1 value: 41.9307 - type: nauc_recall_at_3_max value: 35.8036 - type: nauc_recall_at_3_std value: 14.462 - type: nauc_recall_at_3_diff1 value: 28.1908 - type: nauc_recall_at_5_max value: 31.9315 - type: nauc_recall_at_5_std value: 12.5393 - type: nauc_recall_at_5_diff1 value: 24.0762 - type: nauc_recall_at_10_max value: 26.450200000000002 - type: nauc_recall_at_10_std value: 10.0809 - type: nauc_recall_at_10_diff1 value: 17.9437 - type: nauc_recall_at_20_max value: 22.489600000000003 - type: nauc_recall_at_20_std value: 11.888 - type: nauc_recall_at_20_diff1 value: 13.223199999999999 - type: nauc_recall_at_100_max value: 21.118100000000002 - type: nauc_recall_at_100_std value: 19.320300000000003 - type: nauc_recall_at_100_diff1 value: 7.9007 - type: nauc_recall_at_1000_max value: 15.723 - type: nauc_recall_at_1000_std value: 11.5761 - type: nauc_recall_at_1000_diff1 value: 0.9648 - type: nauc_precision_at_1_max value: 44.858 - type: nauc_precision_at_1_std value: 12.721499999999999 - type: nauc_precision_at_1_diff1 value: 41.9307 - type: nauc_precision_at_3_max value: 35.8036 - type: nauc_precision_at_3_std value: 14.462 - type: nauc_precision_at_3_diff1 value: 28.1908 - type: nauc_precision_at_5_max value: 31.9315 - type: nauc_precision_at_5_std value: 12.5393 - type: nauc_precision_at_5_diff1 value: 24.0762 - type: nauc_precision_at_10_max value: 26.450200000000002 - type: nauc_precision_at_10_std value: 10.0809 - type: nauc_precision_at_10_diff1 value: 17.9437 - type: nauc_precision_at_20_max value: 22.489600000000003 - type: nauc_precision_at_20_std value: 11.888 - type: nauc_precision_at_20_diff1 value: 13.223199999999999 - type: nauc_precision_at_100_max value: 21.118100000000002 - type: nauc_precision_at_100_std value: 19.320300000000003 - type: nauc_precision_at_100_diff1 value: 7.9007 - type: nauc_precision_at_1000_max value: 15.723 - type: nauc_precision_at_1000_std value: 11.5761 - type: nauc_precision_at_1000_diff1 value: 0.9648 - type: nauc_mrr_at_1_max value: 44.858 - type: nauc_mrr_at_1_std value: 12.721499999999999 - type: nauc_mrr_at_1_diff1 value: 41.9307 - type: nauc_mrr_at_3_max value: 40.1323 - type: nauc_mrr_at_3_std value: 13.677200000000001 - type: nauc_mrr_at_3_diff1 value: 34.828399999999995 - type: nauc_mrr_at_5_max value: 38.7125 - type: nauc_mrr_at_5_std value: 13.070300000000001 - type: nauc_mrr_at_5_diff1 value: 33.3574 - type: nauc_mrr_at_10_max value: 37.3599 - type: nauc_mrr_at_10_std value: 12.5138 - type: nauc_mrr_at_10_diff1 value: 31.848599999999998 - type: nauc_mrr_at_20_max value: 36.3682 - type: nauc_mrr_at_20_std value: 12.6613 - type: nauc_mrr_at_20_diff1 value: 30.641000000000002 - type: nauc_mrr_at_100_max value: 35.8721 - type: nauc_mrr_at_100_std value: 13.1938 - type: nauc_mrr_at_100_diff1 value: 29.8344 - type: nauc_mrr_at_1000_max value: 35.7943 - type: nauc_mrr_at_1000_std value: 13.2437 - type: nauc_mrr_at_1000_diff1 value: 29.710199999999997 - type: main_score value: 5.431 task: type: Retrieval - dataset: config: vie-ara name: MTEB MLQARetrieval (vie-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 5.667 - type: ndcg_at_3 value: 8.308 - type: ndcg_at_5 value: 9.481 - type: ndcg_at_10 value: 10.795 - type: ndcg_at_20 value: 11.993 - type: ndcg_at_100 value: 14.754000000000001 - type: ndcg_at_1000 value: 20.329 - type: map_at_1 value: 5.667 - type: map_at_3 value: 7.637 - type: map_at_5 value: 8.292 - type: map_at_10 value: 8.834999999999999 - type: map_at_20 value: 9.164 - type: map_at_100 value: 9.519 - type: map_at_1000 value: 9.681 - type: recall_at_1 value: 5.667 - type: recall_at_3 value: 10.259 - type: recall_at_5 value: 13.092 - type: recall_at_10 value: 17.147000000000002 - type: recall_at_20 value: 21.886 - type: recall_at_100 value: 37.175999999999995 - type: recall_at_1000 value: 83.977 - type: precision_at_1 value: 5.667 - type: precision_at_3 value: 3.42 - type: precision_at_5 value: 2.618 - type: precision_at_10 value: 1.7149999999999999 - type: precision_at_20 value: 1.094 - type: precision_at_100 value: 0.372 - type: precision_at_1000 value: 0.084 - type: mrr_at_1 value: 5.6668 - type: mrr_at_3 value: 7.6372 - type: mrr_at_5 value: 8.2918 - type: mrr_at_10 value: 8.8355 - type: mrr_at_20 value: 9.164 - type: mrr_at_100 value: 9.519400000000001 - type: mrr_at_1000 value: 9.681 - type: nauc_ndcg_at_1_max value: 42.6394 - type: nauc_ndcg_at_1_std value: 27.9872 - type: nauc_ndcg_at_1_diff1 value: 45.9829 - type: nauc_ndcg_at_3_max value: 38.122299999999996 - type: nauc_ndcg_at_3_std value: 23.1287 - type: nauc_ndcg_at_3_diff1 value: 34.5182 - type: nauc_ndcg_at_5_max value: 37.161899999999996 - type: nauc_ndcg_at_5_std value: 24.085 - type: nauc_ndcg_at_5_diff1 value: 29.9462 - type: nauc_ndcg_at_10_max value: 34.996 - type: nauc_ndcg_at_10_std value: 23.091900000000003 - type: nauc_ndcg_at_10_diff1 value: 27.536699999999996 - type: nauc_ndcg_at_20_max value: 34.7649 - type: nauc_ndcg_at_20_std value: 23.7151 - type: nauc_ndcg_at_20_diff1 value: 26.2981 - type: nauc_ndcg_at_100_max value: 34.5203 - type: nauc_ndcg_at_100_std value: 24.5469 - type: nauc_ndcg_at_100_diff1 value: 23.6597 - type: nauc_ndcg_at_1000_max value: 36.7572 - type: nauc_ndcg_at_1000_std value: 26.974999999999998 - type: nauc_ndcg_at_1000_diff1 value: 25.544800000000002 - type: nauc_map_at_1_max value: 42.6394 - type: nauc_map_at_1_std value: 27.9872 - type: nauc_map_at_1_diff1 value: 45.9829 - type: nauc_map_at_3_max value: 39.0702 - type: nauc_map_at_3_std value: 24.1064 - type: nauc_map_at_3_diff1 value: 36.708 - type: nauc_map_at_5_max value: 38.4039 - type: nauc_map_at_5_std value: 24.6638 - type: nauc_map_at_5_diff1 value: 33.6966 - type: nauc_map_at_10_max value: 37.2626 - type: nauc_map_at_10_std value: 24.1767 - type: nauc_map_at_10_diff1 value: 32.3835 - type: nauc_map_at_20_max value: 37.119 - type: nauc_map_at_20_std value: 24.3649 - type: nauc_map_at_20_diff1 value: 31.852200000000003 - type: nauc_map_at_100_max value: 37.1269 - type: nauc_map_at_100_std value: 24.5511 - type: nauc_map_at_100_diff1 value: 31.3411 - type: nauc_map_at_1000_max value: 37.2113 - type: nauc_map_at_1000_std value: 24.6889 - type: nauc_map_at_1000_diff1 value: 31.372600000000002 - type: nauc_recall_at_1_max value: 42.6394 - type: nauc_recall_at_1_std value: 27.9872 - type: nauc_recall_at_1_diff1 value: 45.9829 - type: nauc_recall_at_3_max value: 36.0345 - type: nauc_recall_at_3_std value: 20.9864 - type: nauc_recall_at_3_diff1 value: 29.752499999999998 - type: nauc_recall_at_5_max value: 34.6542 - type: nauc_recall_at_5_std value: 23.0252 - type: nauc_recall_at_5_diff1 value: 22.2777 - type: nauc_recall_at_10_max value: 30.7022 - type: nauc_recall_at_10_std value: 21.087500000000002 - type: nauc_recall_at_10_diff1 value: 18.5945 - type: nauc_recall_at_20_max value: 30.833899999999996 - type: nauc_recall_at_20_std value: 22.8847 - type: nauc_recall_at_20_diff1 value: 16.9343 - type: nauc_recall_at_100_max value: 30.2085 - type: nauc_recall_at_100_std value: 25.151899999999998 - type: nauc_recall_at_100_diff1 value: 10.5572 - type: nauc_recall_at_1000_max value: 45.5105 - type: nauc_recall_at_1000_std value: 45.5239 - type: nauc_recall_at_1000_diff1 value: 13.6761 - type: nauc_precision_at_1_max value: 42.6394 - type: nauc_precision_at_1_std value: 27.9872 - type: nauc_precision_at_1_diff1 value: 45.9829 - type: nauc_precision_at_3_max value: 36.0345 - type: nauc_precision_at_3_std value: 20.9864 - type: nauc_precision_at_3_diff1 value: 29.752499999999998 - type: nauc_precision_at_5_max value: 34.6542 - type: nauc_precision_at_5_std value: 23.0252 - type: nauc_precision_at_5_diff1 value: 22.2777 - type: nauc_precision_at_10_max value: 30.7022 - type: nauc_precision_at_10_std value: 21.087500000000002 - type: nauc_precision_at_10_diff1 value: 18.5945 - type: nauc_precision_at_20_max value: 30.833899999999996 - type: nauc_precision_at_20_std value: 22.8847 - type: nauc_precision_at_20_diff1 value: 16.9343 - type: nauc_precision_at_100_max value: 30.2085 - type: nauc_precision_at_100_std value: 25.151899999999998 - type: nauc_precision_at_100_diff1 value: 10.5572 - type: nauc_precision_at_1000_max value: 45.5105 - type: nauc_precision_at_1000_std value: 45.5239 - type: nauc_precision_at_1000_diff1 value: 13.6761 - type: nauc_mrr_at_1_max value: 42.6394 - type: nauc_mrr_at_1_std value: 27.9872 - type: nauc_mrr_at_1_diff1 value: 45.9829 - type: nauc_mrr_at_3_max value: 39.0702 - type: nauc_mrr_at_3_std value: 24.1064 - type: nauc_mrr_at_3_diff1 value: 36.708 - type: nauc_mrr_at_5_max value: 38.4039 - type: nauc_mrr_at_5_std value: 24.6638 - type: nauc_mrr_at_5_diff1 value: 33.6966 - type: nauc_mrr_at_10_max value: 37.2626 - type: nauc_mrr_at_10_std value: 24.1767 - type: nauc_mrr_at_10_diff1 value: 32.3835 - type: nauc_mrr_at_20_max value: 37.119 - type: nauc_mrr_at_20_std value: 24.3649 - type: nauc_mrr_at_20_diff1 value: 31.852200000000003 - type: nauc_mrr_at_100_max value: 37.1269 - type: nauc_mrr_at_100_std value: 24.5511 - type: nauc_mrr_at_100_diff1 value: 31.3411 - type: nauc_mrr_at_1000_max value: 37.2113 - type: nauc_mrr_at_1000_std value: 24.6889 - type: nauc_mrr_at_1000_diff1 value: 31.372600000000002 - type: main_score value: 10.795 task: type: Retrieval - dataset: config: zho-ara name: MTEB MLQARetrieval (zho-ara) revision: 397ed406c1a7902140303e7faf60fff35b58d285 split: test type: facebook/mlqa metrics: - type: ndcg_at_1 value: 3.977 - type: ndcg_at_3 value: 5.531 - type: ndcg_at_5 value: 6.166 - type: ndcg_at_10 value: 6.815 - type: ndcg_at_20 value: 7.375 - type: ndcg_at_100 value: 9.419 - type: ndcg_at_1000 value: 15.168999999999999 - type: map_at_1 value: 3.977 - type: map_at_3 value: 5.146 - type: map_at_5 value: 5.504 - type: map_at_10 value: 5.768 - type: map_at_20 value: 5.918 - type: map_at_100 value: 6.171 - type: map_at_1000 value: 6.313000000000001 - type: recall_at_1 value: 3.977 - type: recall_at_3 value: 6.646000000000001 - type: recall_at_5 value: 8.163 - type: recall_at_10 value: 10.177999999999999 - type: recall_at_20 value: 12.428 - type: recall_at_100 value: 23.862 - type: recall_at_1000 value: 73.67899999999999 - type: precision_at_1 value: 3.977 - type: precision_at_3 value: 2.215 - type: precision_at_5 value: 1.633 - type: precision_at_10 value: 1.02 - type: precision_at_20 value: 0.623 - type: precision_at_100 value: 0.23900000000000002 - type: precision_at_1000 value: 0.074 - type: mrr_at_1 value: 3.977 - type: mrr_at_3 value: 5.1456 - type: mrr_at_5 value: 5.5041 - type: mrr_at_10 value: 5.7719 - type: mrr_at_20 value: 5.9213000000000005 - type: mrr_at_100 value: 6.1742 - type: mrr_at_1000 value: 6.316199999999999 - type: nauc_ndcg_at_1_max value: 49.7326 - type: nauc_ndcg_at_1_std value: 21.0715 - type: nauc_ndcg_at_1_diff1 value: 66.4455 - type: nauc_ndcg_at_3_max value: 43.7029 - type: nauc_ndcg_at_3_std value: 20.071 - type: nauc_ndcg_at_3_diff1 value: 48.8581 - type: nauc_ndcg_at_5_max value: 40.5439 - type: nauc_ndcg_at_5_std value: 20.206699999999998 - type: nauc_ndcg_at_5_diff1 value: 43.3761 - type: nauc_ndcg_at_10_max value: 38.9703 - type: nauc_ndcg_at_10_std value: 19.8004 - type: nauc_ndcg_at_10_diff1 value: 39.658500000000004 - type: nauc_ndcg_at_20_max value: 37.9017 - type: nauc_ndcg_at_20_std value: 19.5964 - type: nauc_ndcg_at_20_diff1 value: 38.2607 - type: nauc_ndcg_at_100_max value: 34.8103 - type: nauc_ndcg_at_100_std value: 20.6233 - type: nauc_ndcg_at_100_diff1 value: 33.4776 - type: nauc_ndcg_at_1000_max value: 35.1374 - type: nauc_ndcg_at_1000_std value: 21.701500000000003 - type: nauc_ndcg_at_1000_diff1 value: 32.7129 - type: nauc_map_at_1_max value: 49.7326 - type: nauc_map_at_1_std value: 21.0715 - type: nauc_map_at_1_diff1 value: 66.4455 - type: nauc_map_at_3_max value: 44.931599999999996 - type: nauc_map_at_3_std value: 20.4494 - type: nauc_map_at_3_diff1 value: 52.1408 - type: nauc_map_at_5_max value: 42.8222 - type: nauc_map_at_5_std value: 20.494 - type: nauc_map_at_5_diff1 value: 48.4741 - type: nauc_map_at_10_max value: 41.928399999999996 - type: nauc_map_at_10_std value: 20.3206 - type: nauc_map_at_10_diff1 value: 46.400999999999996 - type: nauc_map_at_20_max value: 41.5137 - type: nauc_map_at_20_std value: 20.2243 - type: nauc_map_at_20_diff1 value: 45.845200000000006 - type: nauc_map_at_100_max value: 40.8889 - type: nauc_map_at_100_std value: 20.4168 - type: nauc_map_at_100_diff1 value: 44.7751 - type: nauc_map_at_1000_max value: 40.8818 - type: nauc_map_at_1000_std value: 20.4879 - type: nauc_map_at_1000_diff1 value: 44.6345 - type: nauc_recall_at_1_max value: 49.7326 - type: nauc_recall_at_1_std value: 21.0715 - type: nauc_recall_at_1_diff1 value: 66.4455 - type: nauc_recall_at_3_max value: 40.9239 - type: nauc_recall_at_3_std value: 19.164 - type: nauc_recall_at_3_diff1 value: 41.5215 - type: nauc_recall_at_5_max value: 35.8311 - type: nauc_recall_at_5_std value: 19.624 - type: nauc_recall_at_5_diff1 value: 32.9591 - type: nauc_recall_at_10_max value: 33.5859 - type: nauc_recall_at_10_std value: 18.8055 - type: nauc_recall_at_10_diff1 value: 27.381299999999996 - type: nauc_recall_at_20_max value: 31.6785 - type: nauc_recall_at_20_std value: 18.4961 - type: nauc_recall_at_20_diff1 value: 25.4357 - type: nauc_recall_at_100_max value: 25.8595 - type: nauc_recall_at_100_std value: 21.3998 - type: nauc_recall_at_100_diff1 value: 17.7469 - type: nauc_recall_at_1000_max value: 24.833199999999998 - type: nauc_recall_at_1000_std value: 26.175500000000003 - type: nauc_recall_at_1000_diff1 value: 11.568399999999999 - type: nauc_precision_at_1_max value: 49.7326 - type: nauc_precision_at_1_std value: 21.0715 - type: nauc_precision_at_1_diff1 value: 66.4455 - type: nauc_precision_at_3_max value: 40.9239 - type: nauc_precision_at_3_std value: 19.164 - type: nauc_precision_at_3_diff1 value: 41.5215 - type: nauc_precision_at_5_max value: 35.8311 - type: nauc_precision_at_5_std value: 19.624 - type: nauc_precision_at_5_diff1 value: 32.9591 - type: nauc_precision_at_10_max value: 33.5103 - type: nauc_precision_at_10_std value: 18.7552 - type: nauc_precision_at_10_diff1 value: 27.3599 - type: nauc_precision_at_20_max value: 31.615399999999998 - type: nauc_precision_at_20_std value: 18.4527 - type: nauc_precision_at_20_diff1 value: 25.419999999999998 - type: nauc_precision_at_100_max value: 25.763399999999997 - type: nauc_precision_at_100_std value: 21.3167 - type: nauc_precision_at_100_diff1 value: 17.7239 - type: nauc_precision_at_1000_max value: 24.7088 - type: nauc_precision_at_1000_std value: 26.0459 - type: nauc_precision_at_1000_diff1 value: 11.5617 - type: nauc_mrr_at_1_max value: 49.7326 - type: nauc_mrr_at_1_std value: 21.0715 - type: nauc_mrr_at_1_diff1 value: 66.4455 - type: nauc_mrr_at_3_max value: 44.931599999999996 - type: nauc_mrr_at_3_std value: 20.4494 - type: nauc_mrr_at_3_diff1 value: 52.1408 - type: nauc_mrr_at_5_max value: 42.8222 - type: nauc_mrr_at_5_std value: 20.494 - type: nauc_mrr_at_5_diff1 value: 48.4741 - type: nauc_mrr_at_10_max value: 41.9004 - type: nauc_mrr_at_10_std value: 20.3046 - type: nauc_mrr_at_10_diff1 value: 46.381299999999996 - type: nauc_mrr_at_20_max value: 41.4863 - type: nauc_mrr_at_20_std value: 20.2087 - type: nauc_mrr_at_20_diff1 value: 45.8262 - type: nauc_mrr_at_100_max value: 40.8681 - type: nauc_mrr_at_100_std value: 20.4048 - type: nauc_mrr_at_100_diff1 value: 44.7603 - type: nauc_mrr_at_1000_max value: 40.861 - type: nauc_mrr_at_1000_std value: 20.4759 - type: nauc_mrr_at_1000_diff1 value: 44.6199 - type: main_score value: 6.815 task: type: Retrieval - dataset: config: ar name: MTEB MintakaRetrieval (ar) revision: efa78cc2f74bbcd21eff2261f9e13aebe40b814e split: test type: jinaai/mintakaqa metrics: - type: ndcg_at_1 value: 11.439 - type: ndcg_at_3 value: 16.081 - type: ndcg_at_5 value: 18.007 - type: ndcg_at_10 value: 20.058999999999997 - type: ndcg_at_20 value: 21.807000000000002 - type: ndcg_at_100 value: 24.918000000000003 - type: ndcg_at_1000 value: 29.343999999999998 - type: map_at_1 value: 11.439 - type: map_at_3 value: 14.927000000000001 - type: map_at_5 value: 15.984000000000002 - type: map_at_10 value: 16.82 - type: map_at_20 value: 17.307 - type: map_at_100 value: 17.709 - type: map_at_1000 value: 17.837 - type: recall_at_1 value: 11.439 - type: recall_at_3 value: 19.428 - type: recall_at_5 value: 24.149 - type: recall_at_10 value: 30.549 - type: recall_at_20 value: 37.403999999999996 - type: recall_at_100 value: 54.607000000000006 - type: recall_at_1000 value: 91.78399999999999 - type: precision_at_1 value: 11.439 - type: precision_at_3 value: 6.476 - type: precision_at_5 value: 4.83 - type: precision_at_10 value: 3.055 - type: precision_at_20 value: 1.87 - type: precision_at_100 value: 0.5459999999999999 - type: precision_at_1000 value: 0.092 - type: mrr_at_1 value: 11.4389 - type: mrr_at_3 value: 14.9266 - type: mrr_at_5 value: 15.984300000000001 - type: mrr_at_10 value: 16.8196 - type: mrr_at_20 value: 17.3074 - type: mrr_at_100 value: 17.7093 - type: mrr_at_1000 value: 17.8368 - type: nauc_ndcg_at_1_max value: 20.689799999999998 - type: nauc_ndcg_at_1_std value: 8.8259 - type: nauc_ndcg_at_1_diff1 value: 23.087699999999998 - type: nauc_ndcg_at_3_max value: 20.441000000000003 - type: nauc_ndcg_at_3_std value: 11.9201 - type: nauc_ndcg_at_3_diff1 value: 15.631800000000002 - type: nauc_ndcg_at_5_max value: 20.5375 - type: nauc_ndcg_at_5_std value: 11.7772 - type: nauc_ndcg_at_5_diff1 value: 15.046999999999999 - type: nauc_ndcg_at_10_max value: 20.1703 - type: nauc_ndcg_at_10_std value: 12.2441 - type: nauc_ndcg_at_10_diff1 value: 14.202300000000001 - type: nauc_ndcg_at_20_max value: 19.605900000000002 - type: nauc_ndcg_at_20_std value: 12.899 - type: nauc_ndcg_at_20_diff1 value: 14.9112 - type: nauc_ndcg_at_100_max value: 17.7479 - type: nauc_ndcg_at_100_std value: 13.1804 - type: nauc_ndcg_at_100_diff1 value: 13.7593 - type: nauc_ndcg_at_1000_max value: 18.5596 - type: nauc_ndcg_at_1000_std value: 12.3602 - type: nauc_ndcg_at_1000_diff1 value: 14.955499999999999 - type: nauc_map_at_1_max value: 20.689799999999998 - type: nauc_map_at_1_std value: 8.8259 - type: nauc_map_at_1_diff1 value: 23.087699999999998 - type: nauc_map_at_3_max value: 20.604 - type: nauc_map_at_3_std value: 11.261899999999999 - type: nauc_map_at_3_diff1 value: 17.194699999999997 - type: nauc_map_at_5_max value: 20.6675 - type: nauc_map_at_5_std value: 11.1805 - type: nauc_map_at_5_diff1 value: 16.8416 - type: nauc_map_at_10_max value: 20.4983 - type: nauc_map_at_10_std value: 11.3658 - type: nauc_map_at_10_diff1 value: 16.3996 - type: nauc_map_at_20_max value: 20.3242 - type: nauc_map_at_20_std value: 11.5849 - type: nauc_map_at_20_diff1 value: 16.5979 - type: nauc_map_at_100_max value: 20.052300000000002 - type: nauc_map_at_100_std value: 11.6264 - type: nauc_map_at_100_diff1 value: 16.4362 - type: nauc_map_at_1000_max value: 20.0687 - type: nauc_map_at_1000_std value: 11.606 - type: nauc_map_at_1000_diff1 value: 16.4891 - type: nauc_recall_at_1_max value: 20.689799999999998 - type: nauc_recall_at_1_std value: 8.8259 - type: nauc_recall_at_1_diff1 value: 23.087699999999998 - type: nauc_recall_at_3_max value: 20.0186 - type: nauc_recall_at_3_std value: 13.4853 - type: nauc_recall_at_3_diff1 value: 11.9226 - type: nauc_recall_at_5_max value: 20.2188 - type: nauc_recall_at_5_std value: 13.0968 - type: nauc_recall_at_5_diff1 value: 10.9832 - type: nauc_recall_at_10_max value: 19.3773 - type: nauc_recall_at_10_std value: 14.227400000000001 - type: nauc_recall_at_10_diff1 value: 9.461500000000001 - type: nauc_recall_at_20_max value: 17.729 - type: nauc_recall_at_20_std value: 16.033 - type: nauc_recall_at_20_diff1 value: 11.7912 - type: nauc_recall_at_100_max value: 9.5088 - type: nauc_recall_at_100_std value: 17.4782 - type: nauc_recall_at_100_diff1 value: 6.4486 - type: nauc_recall_at_1000_max value: 6.4045000000000005 - type: nauc_recall_at_1000_std value: 9.5314 - type: nauc_recall_at_1000_diff1 value: 11.1984 - type: nauc_precision_at_1_max value: 20.689799999999998 - type: nauc_precision_at_1_std value: 8.8259 - type: nauc_precision_at_1_diff1 value: 23.087699999999998 - type: nauc_precision_at_3_max value: 20.0186 - type: nauc_precision_at_3_std value: 13.4853 - type: nauc_precision_at_3_diff1 value: 11.9226 - type: nauc_precision_at_5_max value: 20.2188 - type: nauc_precision_at_5_std value: 13.0968 - type: nauc_precision_at_5_diff1 value: 10.9832 - type: nauc_precision_at_10_max value: 19.3773 - type: nauc_precision_at_10_std value: 14.227400000000001 - type: nauc_precision_at_10_diff1 value: 9.461500000000001 - type: nauc_precision_at_20_max value: 17.729 - type: nauc_precision_at_20_std value: 16.033 - type: nauc_precision_at_20_diff1 value: 11.7912 - type: nauc_precision_at_100_max value: 9.5088 - type: nauc_precision_at_100_std value: 17.4782 - type: nauc_precision_at_100_diff1 value: 6.4486 - type: nauc_precision_at_1000_max value: 6.4045000000000005 - type: nauc_precision_at_1000_std value: 9.5314 - type: nauc_precision_at_1000_diff1 value: 11.1984 - type: nauc_mrr_at_1_max value: 20.689799999999998 - type: nauc_mrr_at_1_std value: 8.8259 - type: nauc_mrr_at_1_diff1 value: 23.087699999999998 - type: nauc_mrr_at_3_max value: 20.604 - type: nauc_mrr_at_3_std value: 11.261899999999999 - type: nauc_mrr_at_3_diff1 value: 17.194699999999997 - type: nauc_mrr_at_5_max value: 20.6675 - type: nauc_mrr_at_5_std value: 11.1805 - type: nauc_mrr_at_5_diff1 value: 16.8416 - type: nauc_mrr_at_10_max value: 20.4983 - type: nauc_mrr_at_10_std value: 11.3658 - type: nauc_mrr_at_10_diff1 value: 16.3996 - type: nauc_mrr_at_20_max value: 20.3242 - type: nauc_mrr_at_20_std value: 11.5849 - type: nauc_mrr_at_20_diff1 value: 16.5979 - type: nauc_mrr_at_100_max value: 20.052300000000002 - type: nauc_mrr_at_100_std value: 11.6264 - type: nauc_mrr_at_100_diff1 value: 16.4362 - type: nauc_mrr_at_1000_max value: 20.0687 - type: nauc_mrr_at_1000_std value: 11.606 - type: nauc_mrr_at_1000_diff1 value: 16.4891 - type: main_score value: 20.058999999999997 task: type: Retrieval - dataset: config: arabic name: MTEB MrTidyRetrieval (arabic) revision: fc24a3ce8f09746410daee3d5cd823ff7a0675b7 split: test type: mteb/mrtidy metrics: - type: ndcg_at_1 value: 39.129999999999995 - type: ndcg_at_3 value: 49.21 - type: ndcg_at_5 value: 53.391 - type: ndcg_at_10 value: 56.628 - type: ndcg_at_20 value: 57.952999999999996 - type: ndcg_at_100 value: 59.536 - type: ndcg_at_1000 value: 60.331 - type: map_at_1 value: 36.232 - type: map_at_3 value: 45.900999999999996 - type: map_at_5 value: 48.298 - type: map_at_10 value: 49.748 - type: map_at_20 value: 50.14999999999999 - type: map_at_100 value: 50.388999999999996 - type: map_at_1000 value: 50.422 - type: recall_at_1 value: 36.232 - type: recall_at_3 value: 56.568 - type: recall_at_5 value: 66.374 - type: recall_at_10 value: 75.809 - type: recall_at_20 value: 80.743 - type: recall_at_100 value: 88.86800000000001 - type: recall_at_1000 value: 94.83500000000001 - type: precision_at_1 value: 39.129999999999995 - type: precision_at_3 value: 20.814 - type: precision_at_5 value: 14.764 - type: precision_at_10 value: 8.584999999999999 - type: precision_at_20 value: 4.598 - type: precision_at_100 value: 1.023 - type: precision_at_1000 value: 0.11 - type: mrr_at_1 value: 39.1304 - type: mrr_at_3 value: 48.1653 - type: mrr_at_5 value: 50.4363 - type: mrr_at_10 value: 51.659299999999995 - type: mrr_at_20 value: 51.9948 - type: mrr_at_100 value: 52.168899999999994 - type: mrr_at_1000 value: 52.1923 - type: nauc_ndcg_at_1_max value: 31.0079 - type: nauc_ndcg_at_1_std value: 0.3105 - type: nauc_ndcg_at_1_diff1 value: 45.345 - type: nauc_ndcg_at_3_max value: 34.8656 - type: nauc_ndcg_at_3_std value: 1.379 - type: nauc_ndcg_at_3_diff1 value: 39.2231 - type: nauc_ndcg_at_5_max value: 36.424099999999996 - type: nauc_ndcg_at_5_std value: 1.7812999999999999 - type: nauc_ndcg_at_5_diff1 value: 38.9234 - type: nauc_ndcg_at_10_max value: 39.0858 - type: nauc_ndcg_at_10_std value: 5.3904 - type: nauc_ndcg_at_10_diff1 value: 38.703199999999995 - type: nauc_ndcg_at_20_max value: 38.8658 - type: nauc_ndcg_at_20_std value: 5.8416999999999994 - type: nauc_ndcg_at_20_diff1 value: 38.1471 - type: nauc_ndcg_at_100_max value: 38.798 - type: nauc_ndcg_at_100_std value: 6.5924 - type: nauc_ndcg_at_100_diff1 value: 38.757999999999996 - type: nauc_ndcg_at_1000_max value: 38.2341 - type: nauc_ndcg_at_1000_std value: 6.092899999999999 - type: nauc_ndcg_at_1000_diff1 value: 38.9228 - type: nauc_map_at_1_max value: 28.547299999999996 - type: nauc_map_at_1_std value: -1.7869 - type: nauc_map_at_1_diff1 value: 45.2351 - type: nauc_map_at_3_max value: 33.3973 - type: nauc_map_at_3_std value: 0.527 - type: nauc_map_at_3_diff1 value: 40.855999999999995 - type: nauc_map_at_5_max value: 34.3218 - type: nauc_map_at_5_std value: 0.7775 - type: nauc_map_at_5_diff1 value: 40.8516 - type: nauc_map_at_10_max value: 35.4687 - type: nauc_map_at_10_std value: 2.2894 - type: nauc_map_at_10_diff1 value: 40.7375 - type: nauc_map_at_20_max value: 35.4342 - type: nauc_map_at_20_std value: 2.387 - type: nauc_map_at_20_diff1 value: 40.598099999999995 - type: nauc_map_at_100_max value: 35.436 - type: nauc_map_at_100_std value: 2.4977 - type: nauc_map_at_100_diff1 value: 40.6903 - type: nauc_map_at_1000_max value: 35.4148 - type: nauc_map_at_1000_std value: 2.4848 - type: nauc_map_at_1000_diff1 value: 40.6969 - type: nauc_recall_at_1_max value: 28.547299999999996 - type: nauc_recall_at_1_std value: -1.7869 - type: nauc_recall_at_1_diff1 value: 45.2351 - type: nauc_recall_at_3_max value: 36.9589 - type: nauc_recall_at_3_std value: 2.461 - type: nauc_recall_at_3_diff1 value: 34.0186 - type: nauc_recall_at_5_max value: 40.954800000000006 - type: nauc_recall_at_5_std value: 3.4049000000000005 - type: nauc_recall_at_5_diff1 value: 31.8241 - type: nauc_recall_at_10_max value: 52.6817 - type: nauc_recall_at_10_std value: 18.2999 - type: nauc_recall_at_10_diff1 value: 29.524499999999996 - type: nauc_recall_at_20_max value: 53.911 - type: nauc_recall_at_20_std value: 23.8157 - type: nauc_recall_at_20_diff1 value: 24.4882 - type: nauc_recall_at_100_max value: 62.9626 - type: nauc_recall_at_100_std value: 43.2649 - type: nauc_recall_at_100_diff1 value: 24.2515 - type: nauc_recall_at_1000_max value: 69.389 - type: nauc_recall_at_1000_std value: 62.872499999999995 - type: nauc_recall_at_1000_diff1 value: 15.781 - type: nauc_precision_at_1_max value: 31.0079 - type: nauc_precision_at_1_std value: 0.3105 - type: nauc_precision_at_1_diff1 value: 45.345 - type: nauc_precision_at_3_max value: 38.9808 - type: nauc_precision_at_3_std value: 6.055 - type: nauc_precision_at_3_diff1 value: 30.8585 - type: nauc_precision_at_5_max value: 41.698800000000006 - type: nauc_precision_at_5_std value: 8.2552 - type: nauc_precision_at_5_diff1 value: 26.799 - type: nauc_precision_at_10_max value: 44.1513 - type: nauc_precision_at_10_std value: 20.4239 - type: nauc_precision_at_10_diff1 value: 17.821 - type: nauc_precision_at_20_max value: 40.0385 - type: nauc_precision_at_20_std value: 22.1236 - type: nauc_precision_at_20_diff1 value: 10.937 - type: nauc_precision_at_100_max value: 31.007299999999997 - type: nauc_precision_at_100_std value: 26.544 - type: nauc_precision_at_100_diff1 value: 4.1416 - type: nauc_precision_at_1000_max value: 19.3882 - type: nauc_precision_at_1000_std value: 23.1796 - type: nauc_precision_at_1000_diff1 value: -3.1299 - type: nauc_mrr_at_1_max value: 31.0079 - type: nauc_mrr_at_1_std value: 0.3105 - type: nauc_mrr_at_1_diff1 value: 45.345 - type: nauc_mrr_at_3_max value: 34.7646 - type: nauc_mrr_at_3_std value: 2.0087 - type: nauc_mrr_at_3_diff1 value: 40.2648 - type: nauc_mrr_at_5_max value: 35.428 - type: nauc_mrr_at_5_std value: 2.1826000000000003 - type: nauc_mrr_at_5_diff1 value: 40.117599999999996 - type: nauc_mrr_at_10_max value: 36.2427 - type: nauc_mrr_at_10_std value: 3.3503 - type: nauc_mrr_at_10_diff1 value: 40.122 - type: nauc_mrr_at_20_max value: 36.119099999999996 - type: nauc_mrr_at_20_std value: 3.4359 - type: nauc_mrr_at_20_diff1 value: 39.991 - type: nauc_mrr_at_100_max value: 36.0909 - type: nauc_mrr_at_100_std value: 3.5073 - type: nauc_mrr_at_100_diff1 value: 40.1237 - type: nauc_mrr_at_1000_max value: 36.0755 - type: nauc_mrr_at_1000_std value: 3.4960999999999998 - type: nauc_mrr_at_1000_diff1 value: 40.1282 - type: main_score value: 56.628 task: type: Retrieval - dataset: config: default name: MTEB SadeemQuestionRetrieval (default) revision: 3cb0752b182e5d5d740df547748b06663c8e0bd9 split: test type: sadeem-ai/sadeem-ar-eval-retrieval-questions metrics: - type: ndcg_at_1 value: 29.584 - type: ndcg_at_3 value: 62.35000000000001 - type: ndcg_at_5 value: 63.943000000000005 - type: ndcg_at_10 value: 64.66199999999999 - type: ndcg_at_20 value: 65.131 - type: ndcg_at_100 value: 65.64 - type: ndcg_at_1000 value: 65.842 - type: map_at_1 value: 29.584 - type: map_at_3 value: 53.75 - type: map_at_5 value: 54.643 - type: map_at_10 value: 54.943 - type: map_at_20 value: 55.06999999999999 - type: map_at_100 value: 55.142 - type: map_at_1000 value: 55.15 - type: recall_at_1 value: 29.584 - type: recall_at_3 value: 87.458 - type: recall_at_5 value: 91.288 - type: recall_at_10 value: 93.49 - type: recall_at_20 value: 95.357 - type: recall_at_100 value: 98.085 - type: recall_at_1000 value: 99.665 - type: precision_at_1 value: 29.584 - type: precision_at_3 value: 29.153000000000002 - type: precision_at_5 value: 18.258 - type: precision_at_10 value: 9.349 - type: precision_at_20 value: 4.768 - type: precision_at_100 value: 0.9809999999999999 - type: precision_at_1000 value: 0.1 - type: mrr_at_1 value: 26.9507 - type: mrr_at_3 value: 52.01050000000001 - type: mrr_at_5 value: 52.934400000000004 - type: mrr_at_10 value: 53.289500000000004 - type: mrr_at_20 value: 53.412800000000004 - type: mrr_at_100 value: 53.486900000000006 - type: mrr_at_1000 value: 53.494600000000005 - type: nauc_ndcg_at_1_max value: 12.0356 - type: nauc_ndcg_at_1_std value: 0.5192 - type: nauc_ndcg_at_1_diff1 value: 7.8072 - type: nauc_ndcg_at_3_max value: 30.9446 - type: nauc_ndcg_at_3_std value: 8.2618 - type: nauc_ndcg_at_3_diff1 value: -32.0437 - type: nauc_ndcg_at_5_max value: 28.4963 - type: nauc_ndcg_at_5_std value: 8.352500000000001 - type: nauc_ndcg_at_5_diff1 value: -24.2261 - type: nauc_ndcg_at_10_max value: 26.7837 - type: nauc_ndcg_at_10_std value: 8.184 - type: nauc_ndcg_at_10_diff1 value: -20.9816 - type: nauc_ndcg_at_20_max value: 25.446999999999996 - type: nauc_ndcg_at_20_std value: 8.0537 - type: nauc_ndcg_at_20_diff1 value: -17.9894 - type: nauc_ndcg_at_100_max value: 23.999000000000002 - type: nauc_ndcg_at_100_std value: 7.388599999999999 - type: nauc_ndcg_at_100_diff1 value: -15.310000000000002 - type: nauc_ndcg_at_1000_max value: 23.384900000000002 - type: nauc_ndcg_at_1000_std value: 6.937500000000001 - type: nauc_ndcg_at_1000_diff1 value: -14.413 - type: nauc_map_at_1_max value: 12.0356 - type: nauc_map_at_1_std value: 0.5192 - type: nauc_map_at_1_diff1 value: 7.8072 - type: nauc_map_at_3_max value: 23.621100000000002 - type: nauc_map_at_3_std value: 5.832 - type: nauc_map_at_3_diff1 value: -16.857 - type: nauc_map_at_5_max value: 22.264200000000002 - type: nauc_map_at_5_std value: 5.8248 - type: nauc_map_at_5_diff1 value: -12.7098 - type: nauc_map_at_10_max value: 21.603 - type: nauc_map_at_10_std value: 5.7261 - type: nauc_map_at_10_diff1 value: -11.472 - type: nauc_map_at_20_max value: 21.2834 - type: nauc_map_at_20_std value: 5.679 - type: nauc_map_at_20_diff1 value: -10.7725 - type: nauc_map_at_100_max value: 21.1126 - type: nauc_map_at_100_std value: 5.6156 - type: nauc_map_at_100_diff1 value: -10.452200000000001 - type: nauc_map_at_1000_max value: 21.093600000000002 - type: nauc_map_at_1000_std value: 5.602 - type: nauc_map_at_1000_diff1 value: -10.4237 - type: nauc_recall_at_1_max value: 12.0356 - type: nauc_recall_at_1_std value: 0.5192 - type: nauc_recall_at_1_diff1 value: 7.8072 - type: nauc_recall_at_3_max value: 78.15950000000001 - type: nauc_recall_at_3_std value: 23.515800000000002 - type: nauc_recall_at_3_diff1 value: -129.77339999999998 - type: nauc_recall_at_5_max value: 82.3486 - type: nauc_recall_at_5_std value: 29.8504 - type: nauc_recall_at_5_diff1 value: -122.74640000000001 - type: nauc_recall_at_10_max value: 83.8657 - type: nauc_recall_at_10_std value: 35.686099999999996 - type: nauc_recall_at_10_diff1 value: -124.2228 - type: nauc_recall_at_20_max value: 83.6609 - type: nauc_recall_at_20_std value: 44.7232 - type: nauc_recall_at_20_diff1 value: -113.68180000000001 - type: nauc_recall_at_100_max value: 86.18249999999999 - type: nauc_recall_at_100_std value: 56.831900000000005 - type: nauc_recall_at_100_diff1 value: -102.1147 - type: nauc_recall_at_1000_max value: 96.265 - type: nauc_recall_at_1000_std value: 81.02319999999999 - type: nauc_recall_at_1000_diff1 value: -103.0485 - type: nauc_precision_at_1_max value: 12.0356 - type: nauc_precision_at_1_std value: 0.5192 - type: nauc_precision_at_1_diff1 value: 7.8072 - type: nauc_precision_at_3_max value: 78.15950000000001 - type: nauc_precision_at_3_std value: 23.515800000000002 - type: nauc_precision_at_3_diff1 value: -129.77339999999998 - type: nauc_precision_at_5_max value: 82.3486 - type: nauc_precision_at_5_std value: 29.8504 - type: nauc_precision_at_5_diff1 value: -122.74640000000001 - type: nauc_precision_at_10_max value: 83.8657 - type: nauc_precision_at_10_std value: 35.686099999999996 - type: nauc_precision_at_10_diff1 value: -124.2228 - type: nauc_precision_at_20_max value: 83.6609 - type: nauc_precision_at_20_std value: 44.7232 - type: nauc_precision_at_20_diff1 value: -113.68180000000001 - type: nauc_precision_at_100_max value: 86.18249999999999 - type: nauc_precision_at_100_std value: 56.831900000000005 - type: nauc_precision_at_100_diff1 value: -102.1147 - type: nauc_precision_at_1000_max value: 96.265 - type: nauc_precision_at_1000_std value: 81.02319999999999 - type: nauc_precision_at_1000_diff1 value: -103.0485 - type: nauc_mrr_at_1_max value: 9.5515 - type: nauc_mrr_at_1_std value: 2.0859 - type: nauc_mrr_at_1_diff1 value: -18.191 - type: nauc_mrr_at_3_max value: 22.731199999999998 - type: nauc_mrr_at_3_std value: 6.1841 - type: nauc_mrr_at_3_diff1 value: -40.003899999999994 - type: nauc_mrr_at_5_max value: 21.2077 - type: nauc_mrr_at_5_std value: 6.1814 - type: nauc_mrr_at_5_diff1 value: -36.348200000000006 - type: nauc_mrr_at_10_max value: 20.3788 - type: nauc_mrr_at_10_std value: 6.157 - type: nauc_mrr_at_10_diff1 value: -35.098 - type: nauc_mrr_at_20_max value: 20.0609 - type: nauc_mrr_at_20_std value: 6.1335 - type: nauc_mrr_at_20_diff1 value: -34.5488 - type: nauc_mrr_at_100_max value: 19.8761 - type: nauc_mrr_at_100_std value: 6.0599 - type: nauc_mrr_at_100_diff1 value: -34.2843 - type: nauc_mrr_at_1000_max value: 19.856199999999998 - type: nauc_mrr_at_1000_std value: 6.0462 - type: nauc_mrr_at_1000_diff1 value: -34.2626 - type: main_score value: 64.66199999999999 task: type: Retrieval - dataset: config: ara-ara name: MTEB XPQARetrieval (ara-ara) revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f split: test type: jinaai/xpqa metrics: - type: ndcg_at_1 value: 30.4 - type: ndcg_at_3 value: 32.501000000000005 - type: ndcg_at_5 value: 34.305 - type: ndcg_at_10 value: 38.361000000000004 - type: ndcg_at_20 value: 41.543 - type: ndcg_at_100 value: 46.025 - type: ndcg_at_1000 value: 48.647 - type: map_at_1 value: 16.651 - type: map_at_3 value: 26.563 - type: map_at_5 value: 29.805999999999997 - type: map_at_10 value: 32.176 - type: map_at_20 value: 33.337 - type: map_at_100 value: 34.138000000000005 - type: map_at_1000 value: 34.266999999999996 - type: recall_at_1 value: 16.651 - type: recall_at_3 value: 31.316 - type: recall_at_5 value: 38.702 - type: recall_at_10 value: 49.036 - type: recall_at_20 value: 59.636 - type: recall_at_100 value: 80.398 - type: recall_at_1000 value: 98.736 - type: precision_at_1 value: 30.4 - type: precision_at_3 value: 22.489 - type: precision_at_5 value: 17.173 - type: precision_at_10 value: 10.84 - type: precision_at_20 value: 6.447 - type: precision_at_100 value: 1.661 - type: precision_at_1000 value: 0.198 - type: mrr_at_1 value: 30.4 - type: mrr_at_3 value: 36.3333 - type: mrr_at_5 value: 37.7133 - type: mrr_at_10 value: 38.7857 - type: mrr_at_20 value: 39.4511 - type: mrr_at_100 value: 39.9401 - type: mrr_at_1000 value: 39.9951 - type: nauc_ndcg_at_1_max value: 40.9998 - type: nauc_ndcg_at_1_std value: -2.2245 - type: nauc_ndcg_at_1_diff1 value: 33.027 - type: nauc_ndcg_at_3_max value: 35.370400000000004 - type: nauc_ndcg_at_3_std value: -4.5181 - type: nauc_ndcg_at_3_diff1 value: 28.095399999999998 - type: nauc_ndcg_at_5_max value: 34.0099 - type: nauc_ndcg_at_5_std value: -5.1435 - type: nauc_ndcg_at_5_diff1 value: 28.9273 - type: nauc_ndcg_at_10_max value: 36.5137 - type: nauc_ndcg_at_10_std value: -5.0572 - type: nauc_ndcg_at_10_diff1 value: 29.5612 - type: nauc_ndcg_at_20_max value: 36.742799999999995 - type: nauc_ndcg_at_20_std value: -4.5502 - type: nauc_ndcg_at_20_diff1 value: 28.564 - type: nauc_ndcg_at_100_max value: 37.6185 - type: nauc_ndcg_at_100_std value: -1.4362 - type: nauc_ndcg_at_100_diff1 value: 27.787699999999997 - type: nauc_ndcg_at_1000_max value: 37.3053 - type: nauc_ndcg_at_1000_std value: -2.7678000000000003 - type: nauc_ndcg_at_1000_diff1 value: 28.254800000000003 - type: nauc_map_at_1_max value: 21.9573 - type: nauc_map_at_1_std value: -3.9061 - type: nauc_map_at_1_diff1 value: 37.1588 - type: nauc_map_at_3_max value: 30.5096 - type: nauc_map_at_3_std value: -5.1526000000000005 - type: nauc_map_at_3_diff1 value: 29.7545 - type: nauc_map_at_5_max value: 33.0764 - type: nauc_map_at_5_std value: -5.451099999999999 - type: nauc_map_at_5_diff1 value: 29.245500000000003 - type: nauc_map_at_10_max value: 34.8054 - type: nauc_map_at_10_std value: -5.6552999999999995 - type: nauc_map_at_10_diff1 value: 29.326 - type: nauc_map_at_20_max value: 35.1184 - type: nauc_map_at_20_std value: -5.5455 - type: nauc_map_at_20_diff1 value: 29.066599999999998 - type: nauc_map_at_100_max value: 35.225699999999996 - type: nauc_map_at_100_std value: -5.0046 - type: nauc_map_at_100_diff1 value: 28.903000000000002 - type: nauc_map_at_1000_max value: 35.2286 - type: nauc_map_at_1000_std value: -5.0279 - type: nauc_map_at_1000_diff1 value: 28.9084 - type: nauc_recall_at_1_max value: 21.9573 - type: nauc_recall_at_1_std value: -3.9061 - type: nauc_recall_at_1_diff1 value: 37.1588 - type: nauc_recall_at_3_max value: 26.9576 - type: nauc_recall_at_3_std value: -5.1337 - type: nauc_recall_at_3_diff1 value: 25.398799999999998 - type: nauc_recall_at_5_max value: 28.874 - type: nauc_recall_at_5_std value: -6.0468 - type: nauc_recall_at_5_diff1 value: 25.576700000000002 - type: nauc_recall_at_10_max value: 34.2603 - type: nauc_recall_at_10_std value: -5.3883 - type: nauc_recall_at_10_diff1 value: 27.351 - type: nauc_recall_at_20_max value: 33.6132 - type: nauc_recall_at_20_std value: -3.6536 - type: nauc_recall_at_20_diff1 value: 23.3506 - type: nauc_recall_at_100_max value: 39.2982 - type: nauc_recall_at_100_std value: 20.0323 - type: nauc_recall_at_100_diff1 value: 16.1223 - type: nauc_recall_at_1000_max value: 42.5269 - type: nauc_recall_at_1000_std value: 54.9326 - type: nauc_recall_at_1000_diff1 value: -14.3299 - type: nauc_precision_at_1_max value: 40.9998 - type: nauc_precision_at_1_std value: -2.2245 - type: nauc_precision_at_1_diff1 value: 33.027 - type: nauc_precision_at_3_max value: 41.261199999999995 - type: nauc_precision_at_3_std value: -2.6347 - type: nauc_precision_at_3_diff1 value: 17.2095 - type: nauc_precision_at_5_max value: 40.4626 - type: nauc_precision_at_5_std value: -2.6712 - type: nauc_precision_at_5_diff1 value: 15.3469 - type: nauc_precision_at_10_max value: 40.102900000000005 - type: nauc_precision_at_10_std value: -2.8346 - type: nauc_precision_at_10_diff1 value: 14.1092 - type: nauc_precision_at_20_max value: 38.231700000000004 - type: nauc_precision_at_20_std value: -1.1757 - type: nauc_precision_at_20_diff1 value: 10.448699999999999 - type: nauc_precision_at_100_max value: 35.451 - type: nauc_precision_at_100_std value: 6.1677 - type: nauc_precision_at_100_diff1 value: 3.9267999999999996 - type: nauc_precision_at_1000_max value: 28.150399999999998 - type: nauc_precision_at_1000_std value: 2.6082 - type: nauc_precision_at_1000_diff1 value: -0.5528 - type: nauc_mrr_at_1_max value: 40.9998 - type: nauc_mrr_at_1_std value: -2.2245 - type: nauc_mrr_at_1_diff1 value: 33.027 - type: nauc_mrr_at_3_max value: 39.1088 - type: nauc_mrr_at_3_std value: -2.6206 - type: nauc_mrr_at_3_diff1 value: 30.753700000000002 - type: nauc_mrr_at_5_max value: 38.699099999999994 - type: nauc_mrr_at_5_std value: -2.9742 - type: nauc_mrr_at_5_diff1 value: 30.5761 - type: nauc_mrr_at_10_max value: 39.2778 - type: nauc_mrr_at_10_std value: -2.8316999999999997 - type: nauc_mrr_at_10_diff1 value: 31.053399999999996 - type: nauc_mrr_at_20_max value: 39.158 - type: nauc_mrr_at_20_std value: -2.7522 - type: nauc_mrr_at_20_diff1 value: 30.721799999999998 - type: nauc_mrr_at_100_max value: 39.204299999999996 - type: nauc_mrr_at_100_std value: -2.4625 - type: nauc_mrr_at_100_diff1 value: 30.6537 - type: nauc_mrr_at_1000_max value: 39.205 - type: nauc_mrr_at_1000_std value: -2.4861 - type: nauc_mrr_at_1000_diff1 value: 30.6691 - type: main_score value: 38.361000000000004 task: type: Retrieval - dataset: config: eng-ara name: MTEB XPQARetrieval (eng-ara) revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f split: test type: jinaai/xpqa metrics: - type: ndcg_at_1 value: 8.267 - type: ndcg_at_3 value: 7.963000000000001 - type: ndcg_at_5 value: 8.688 - type: ndcg_at_10 value: 10.541 - type: ndcg_at_20 value: 12.478 - type: ndcg_at_100 value: 17.335 - type: ndcg_at_1000 value: 24.310000000000002 - type: map_at_1 value: 3.887 - type: map_at_3 value: 5.887 - type: map_at_5 value: 6.589 - type: map_at_10 value: 7.449999999999999 - type: map_at_20 value: 8.062 - type: map_at_100 value: 8.785 - type: map_at_1000 value: 9.07 - type: recall_at_1 value: 3.887 - type: recall_at_3 value: 7.856000000000001 - type: recall_at_5 value: 10.16 - type: recall_at_10 value: 14.887 - type: recall_at_20 value: 21.163999999999998 - type: recall_at_100 value: 43.789 - type: recall_at_1000 value: 91.229 - type: precision_at_1 value: 8.267 - type: precision_at_3 value: 5.378 - type: precision_at_5 value: 4.373 - type: precision_at_10 value: 3.227 - type: precision_at_20 value: 2.287 - type: precision_at_100 value: 0.9079999999999999 - type: precision_at_1000 value: 0.191 - type: mrr_at_1 value: 8.2667 - type: mrr_at_3 value: 10.8 - type: mrr_at_5 value: 11.6133 - type: mrr_at_10 value: 12.3235 - type: mrr_at_20 value: 12.797600000000001 - type: mrr_at_100 value: 13.4024 - type: mrr_at_1000 value: 13.56 - type: nauc_ndcg_at_1_max value: 17.1915 - type: nauc_ndcg_at_1_std value: 1.7163000000000002 - type: nauc_ndcg_at_1_diff1 value: 25.4884 - type: nauc_ndcg_at_3_max value: 18.8765 - type: nauc_ndcg_at_3_std value: 1.8692 - type: nauc_ndcg_at_3_diff1 value: 23.0826 - type: nauc_ndcg_at_5_max value: 21.1176 - type: nauc_ndcg_at_5_std value: 3.3050999999999995 - type: nauc_ndcg_at_5_diff1 value: 23.0832 - type: nauc_ndcg_at_10_max value: 23.1393 - type: nauc_ndcg_at_10_std value: 4.4501 - type: nauc_ndcg_at_10_diff1 value: 20.1578 - type: nauc_ndcg_at_20_max value: 23.2406 - type: nauc_ndcg_at_20_std value: 4.3849 - type: nauc_ndcg_at_20_diff1 value: 18.214 - type: nauc_ndcg_at_100_max value: 24.317800000000002 - type: nauc_ndcg_at_100_std value: 9.1429 - type: nauc_ndcg_at_100_diff1 value: 16.6206 - type: nauc_ndcg_at_1000_max value: 23.0215 - type: nauc_ndcg_at_1000_std value: 9.1975 - type: nauc_ndcg_at_1000_diff1 value: 17.6542 - type: nauc_map_at_1_max value: 28.0319 - type: nauc_map_at_1_std value: 5.257 - type: nauc_map_at_1_diff1 value: 38.333600000000004 - type: nauc_map_at_3_max value: 22.020200000000003 - type: nauc_map_at_3_std value: 2.7072 - type: nauc_map_at_3_diff1 value: 27.944999999999997 - type: nauc_map_at_5_max value: 23.403299999999998 - type: nauc_map_at_5_std value: 3.5206 - type: nauc_map_at_5_diff1 value: 26.678800000000003 - type: nauc_map_at_10_max value: 24.5692 - type: nauc_map_at_10_std value: 4.3364 - type: nauc_map_at_10_diff1 value: 25.067800000000002 - type: nauc_map_at_20_max value: 24.7846 - type: nauc_map_at_20_std value: 4.5625 - type: nauc_map_at_20_diff1 value: 24.1321 - type: nauc_map_at_100_max value: 25.008799999999997 - type: nauc_map_at_100_std value: 5.6661 - type: nauc_map_at_100_diff1 value: 23.439799999999998 - type: nauc_map_at_1000_max value: 25.0404 - type: nauc_map_at_1000_std value: 5.8388 - type: nauc_map_at_1000_diff1 value: 23.448 - type: nauc_recall_at_1_max value: 28.0319 - type: nauc_recall_at_1_std value: 5.257 - type: nauc_recall_at_1_diff1 value: 38.333600000000004 - type: nauc_recall_at_3_max value: 18.0697 - type: nauc_recall_at_3_std value: 1.4385999999999999 - type: nauc_recall_at_3_diff1 value: 21.2441 - type: nauc_recall_at_5_max value: 20.414199999999997 - type: nauc_recall_at_5_std value: 3.0348 - type: nauc_recall_at_5_diff1 value: 18.8057 - type: nauc_recall_at_10_max value: 23.261599999999998 - type: nauc_recall_at_10_std value: 4.1921 - type: nauc_recall_at_10_diff1 value: 13.2874 - type: nauc_recall_at_20_max value: 23.194 - type: nauc_recall_at_20_std value: 3.7586 - type: nauc_recall_at_20_diff1 value: 9.902800000000001 - type: nauc_recall_at_100_max value: 25.746799999999997 - type: nauc_recall_at_100_std value: 17.0114 - type: nauc_recall_at_100_diff1 value: 6.815200000000001 - type: nauc_recall_at_1000_max value: 29.564 - type: nauc_recall_at_1000_std value: 43.161100000000005 - type: nauc_recall_at_1000_diff1 value: 11.0256 - type: nauc_precision_at_1_max value: 17.1915 - type: nauc_precision_at_1_std value: 1.7163000000000002 - type: nauc_precision_at_1_diff1 value: 25.4884 - type: nauc_precision_at_3_max value: 18.493000000000002 - type: nauc_precision_at_3_std value: 3.8023000000000002 - type: nauc_precision_at_3_diff1 value: 19.173299999999998 - type: nauc_precision_at_5_max value: 20.5674 - type: nauc_precision_at_5_std value: 4.832 - type: nauc_precision_at_5_diff1 value: 17.6175 - type: nauc_precision_at_10_max value: 24.5891 - type: nauc_precision_at_10_std value: 8.334800000000001 - type: nauc_precision_at_10_diff1 value: 13.6769 - type: nauc_precision_at_20_max value: 21.4878 - type: nauc_precision_at_20_std value: 6.8237000000000005 - type: nauc_precision_at_20_diff1 value: 10.6867 - type: nauc_precision_at_100_max value: 18.999299999999998 - type: nauc_precision_at_100_std value: 11.9751 - type: nauc_precision_at_100_diff1 value: 6.9409 - type: nauc_precision_at_1000_max value: -0.6521 - type: nauc_precision_at_1000_std value: 1.4853999999999998 - type: nauc_precision_at_1000_diff1 value: -0.3306 - type: nauc_mrr_at_1_max value: 17.1915 - type: nauc_mrr_at_1_std value: 1.7163000000000002 - type: nauc_mrr_at_1_diff1 value: 25.4884 - type: nauc_mrr_at_3_max value: 16.0442 - type: nauc_mrr_at_3_std value: 2.233 - type: nauc_mrr_at_3_diff1 value: 21.0047 - type: nauc_mrr_at_5_max value: 16.2164 - type: nauc_mrr_at_5_std value: 2.3148 - type: nauc_mrr_at_5_diff1 value: 20.532500000000002 - type: nauc_mrr_at_10_max value: 16.8426 - type: nauc_mrr_at_10_std value: 2.5132000000000003 - type: nauc_mrr_at_10_diff1 value: 19.180500000000002 - type: nauc_mrr_at_20_max value: 16.985500000000002 - type: nauc_mrr_at_20_std value: 2.5334 - type: nauc_mrr_at_20_diff1 value: 18.6229 - type: nauc_mrr_at_100_max value: 17.0516 - type: nauc_mrr_at_100_std value: 3.0317 - type: nauc_mrr_at_100_diff1 value: 18.599 - type: nauc_mrr_at_1000_max value: 17.092399999999998 - type: nauc_mrr_at_1000_std value: 3.0928 - type: nauc_mrr_at_1000_diff1 value: 18.6276 - type: main_score value: 10.541 task: type: Retrieval - dataset: config: ara-eng name: MTEB XPQARetrieval (ara-eng) revision: c99d599f0a6ab9b85b065da6f9d94f9cf731679f split: test type: jinaai/xpqa metrics: - type: ndcg_at_1 value: 6.873 - type: ndcg_at_3 value: 8.016 - type: ndcg_at_5 value: 9.028 - type: ndcg_at_10 value: 11.08 - type: ndcg_at_20 value: 12.83 - type: ndcg_at_100 value: 17.223 - type: ndcg_at_1000 value: 24.052 - type: map_at_1 value: 3.4389999999999996 - type: map_at_3 value: 5.937 - type: map_at_5 value: 7.069 - type: map_at_10 value: 8.092 - type: map_at_20 value: 8.686 - type: map_at_100 value: 9.363000000000001 - type: map_at_1000 value: 9.621 - type: recall_at_1 value: 3.4389999999999996 - type: recall_at_3 value: 8.169 - type: recall_at_5 value: 11.285 - type: recall_at_10 value: 16.497999999999998 - type: recall_at_20 value: 22.051000000000002 - type: recall_at_100 value: 42.284 - type: recall_at_1000 value: 89.996 - type: precision_at_1 value: 6.873 - type: precision_at_3 value: 5.84 - type: precision_at_5 value: 4.879 - type: precision_at_10 value: 3.558 - type: precision_at_20 value: 2.426 - type: precision_at_100 value: 0.902 - type: precision_at_1000 value: 0.185 - type: mrr_at_1 value: 6.8733 - type: mrr_at_3 value: 9.8383 - type: mrr_at_5 value: 10.4111 - type: mrr_at_10 value: 11.186300000000001 - type: mrr_at_20 value: 11.613999999999999 - type: mrr_at_100 value: 12.1222 - type: mrr_at_1000 value: 12.2844 - type: nauc_ndcg_at_1_max value: 38.491 - type: nauc_ndcg_at_1_std value: 19.5273 - type: nauc_ndcg_at_1_diff1 value: 25.796200000000002 - type: nauc_ndcg_at_3_max value: 39.333800000000004 - type: nauc_ndcg_at_3_std value: 20.467 - type: nauc_ndcg_at_3_diff1 value: 24.0404 - type: nauc_ndcg_at_5_max value: 40.8129 - type: nauc_ndcg_at_5_std value: 22.1406 - type: nauc_ndcg_at_5_diff1 value: 21.9089 - type: nauc_ndcg_at_10_max value: 40.1928 - type: nauc_ndcg_at_10_std value: 22.1609 - type: nauc_ndcg_at_10_diff1 value: 17.4233 - type: nauc_ndcg_at_20_max value: 37.9104 - type: nauc_ndcg_at_20_std value: 21.3417 - type: nauc_ndcg_at_20_diff1 value: 16.6991 - type: nauc_ndcg_at_100_max value: 36.626599999999996 - type: nauc_ndcg_at_100_std value: 21.7144 - type: nauc_ndcg_at_100_diff1 value: 14.649799999999999 - type: nauc_ndcg_at_1000_max value: 37.4231 - type: nauc_ndcg_at_1000_std value: 22.2133 - type: nauc_ndcg_at_1000_diff1 value: 16.1937 - type: nauc_map_at_1_max value: 33.697 - type: nauc_map_at_1_std value: 13.289100000000001 - type: nauc_map_at_1_diff1 value: 38.4988 - type: nauc_map_at_3_max value: 38.2279 - type: nauc_map_at_3_std value: 18.9577 - type: nauc_map_at_3_diff1 value: 28.1295 - type: nauc_map_at_5_max value: 41.2897 - type: nauc_map_at_5_std value: 21.786 - type: nauc_map_at_5_diff1 value: 24.9917 - type: nauc_map_at_10_max value: 41.9587 - type: nauc_map_at_10_std value: 22.6477 - type: nauc_map_at_10_diff1 value: 21.868399999999998 - type: nauc_map_at_20_max value: 40.9131 - type: nauc_map_at_20_std value: 22.2096 - type: nauc_map_at_20_diff1 value: 21.1665 - type: nauc_map_at_100_max value: 40.4658 - type: nauc_map_at_100_std value: 22.1678 - type: nauc_map_at_100_diff1 value: 20.3813 - type: nauc_map_at_1000_max value: 40.4857 - type: nauc_map_at_1000_std value: 22.2305 - type: nauc_map_at_1000_diff1 value: 20.4491 - type: nauc_recall_at_1_max value: 33.697 - type: nauc_recall_at_1_std value: 13.289100000000001 - type: nauc_recall_at_1_diff1 value: 38.4988 - type: nauc_recall_at_3_max value: 36.9405 - type: nauc_recall_at_3_std value: 18.470200000000002 - type: nauc_recall_at_3_diff1 value: 22.7851 - type: nauc_recall_at_5_max value: 39.3455 - type: nauc_recall_at_5_std value: 21.6116 - type: nauc_recall_at_5_diff1 value: 18.964100000000002 - type: nauc_recall_at_10_max value: 37.465199999999996 - type: nauc_recall_at_10_std value: 21.142500000000002 - type: nauc_recall_at_10_diff1 value: 11.331900000000001 - type: nauc_recall_at_20_max value: 33.0641 - type: nauc_recall_at_20_std value: 19.5574 - type: nauc_recall_at_20_diff1 value: 11.6818 - type: nauc_recall_at_100_max value: 30.580099999999998 - type: nauc_recall_at_100_std value: 21.1679 - type: nauc_recall_at_100_diff1 value: 7.514600000000001 - type: nauc_recall_at_1000_max value: 38.9651 - type: nauc_recall_at_1000_std value: 33.239000000000004 - type: nauc_recall_at_1000_diff1 value: 14.1121 - type: nauc_precision_at_1_max value: 38.491 - type: nauc_precision_at_1_std value: 19.5273 - type: nauc_precision_at_1_diff1 value: 25.796200000000002 - type: nauc_precision_at_3_max value: 46.392 - type: nauc_precision_at_3_std value: 27.317700000000002 - type: nauc_precision_at_3_diff1 value: 17.732899999999997 - type: nauc_precision_at_5_max value: 48.1446 - type: nauc_precision_at_5_std value: 29.802899999999998 - type: nauc_precision_at_5_diff1 value: 12.6992 - type: nauc_precision_at_10_max value: 45.4786 - type: nauc_precision_at_10_std value: 27.899400000000004 - type: nauc_precision_at_10_diff1 value: 8.1505 - type: nauc_precision_at_20_max value: 36.7663 - type: nauc_precision_at_20_std value: 22.9475 - type: nauc_precision_at_20_diff1 value: 6.2974000000000006 - type: nauc_precision_at_100_max value: 28.255999999999997 - type: nauc_precision_at_100_std value: 20.0854 - type: nauc_precision_at_100_diff1 value: 2.6887000000000003 - type: nauc_precision_at_1000_max value: 13.4244 - type: nauc_precision_at_1000_std value: 9.6022 - type: nauc_precision_at_1000_diff1 value: 0.619 - type: nauc_mrr_at_1_max value: 38.491 - type: nauc_mrr_at_1_std value: 19.5273 - type: nauc_mrr_at_1_diff1 value: 25.796200000000002 - type: nauc_mrr_at_3_max value: 37.8761 - type: nauc_mrr_at_3_std value: 20.2832 - type: nauc_mrr_at_3_diff1 value: 21.5588 - type: nauc_mrr_at_5_max value: 37.8006 - type: nauc_mrr_at_5_std value: 20.660600000000002 - type: nauc_mrr_at_5_diff1 value: 20.7149 - type: nauc_mrr_at_10_max value: 36.713 - type: nauc_mrr_at_10_std value: 20.0126 - type: nauc_mrr_at_10_diff1 value: 18.834 - type: nauc_mrr_at_20_max value: 36.3243 - type: nauc_mrr_at_20_std value: 20.0694 - type: nauc_mrr_at_20_diff1 value: 18.7381 - type: nauc_mrr_at_100_max value: 36.254599999999996 - type: nauc_mrr_at_100_std value: 19.9987 - type: nauc_mrr_at_100_diff1 value: 18.6121 - type: nauc_mrr_at_1000_max value: 36.3176 - type: nauc_mrr_at_1000_std value: 20.0516 - type: nauc_mrr_at_1000_diff1 value: 18.651899999999998 - type: main_score value: 11.08 task: type: Retrieval - dataset: config: ar-ar name: MTEB STS17 (ar-ar) revision: faeb762787bd10488a50c8b5be4a3b82e411949c split: test type: mteb/sts17-crosslingual-sts metrics: - type: cosine_pearson value: 84.66883392015258 - type: cosine_spearman value: 85.30520907141938 - type: euclidean_pearson value: 82.04306779342852 - type: euclidean_spearman value: 84.58744201847996 - type: main_score value: 85.30520907141938 - type: manhattan_pearson value: 82.08829357724328 - type: manhattan_spearman value: 84.49254541383544 task: type: STS license: apache-2.0 --- # Arabic-Triplet-Matryoshka-V2-Model - This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [aubmindlab/bert-base-arabertv02](https://huggingface.co/aubmindlab/bert-base-arabertv02). - It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. - This model is trained on 1M samples from the [akhooli/arabic-triplets-1m-curated-sims-len](https://huggingface.co/datasets/akhooli/arabic-triplets-1m-curated-sims-len) dataset. - Trained for 3 epochs, with final training loss of 0.718 (using MatryoshkaLoss). ```markdown ## Citation If you use the Arabic Matryoshka Embeddings Model, please cite it as follows: @misc{nacar2024enhancingsemanticsimilarityunderstanding, title={Enhancing Semantic Similarity Understanding in Arabic NLP with Nested Embedding Learning}, author={Omer Nacar and Anis Koubaa}, year={2024}, eprint={2407.21139}, archivePrefix={arXiv}, primaryClass={cs.CL}, url={https://arxiv.org/abs/2407.21139}, }