Hugging Face's logo

silma-ai
/
silma-embeddding-sts-v0.1

Sentence Similarity
sentence-transformers
Safetensors
Arabic
English
bert
feature-extraction
Generated from Trainer
loss:CosineSimilarityLoss
mteb
Eval Results
text-embeddings-inference
Inference Endpoints
Model card Files Community
1
bakrianoo commited on
Commit
57d0734
•
1 Parent(s): 869356e

Update Eval Results

Browse files
Files changed (1) hide show
  1. README.md +36 -99
README.md CHANGED
@@ -10,8 +10,6 @@ metrics:
10
  - spearman_euclidean
11
  - pearson_dot
12
  - spearman_dot
13
- - pearson_max
14
- - spearman_max
15
  pipeline_tag: sentence-similarity
16
  tags:
17
  - sentence-transformers
@@ -26,76 +24,70 @@ model-index:
26
  type: semantic-similarity
27
  name: Semantic Similarity
28
  dataset:
29
- name: sts dev 512
30
- type: sts-dev-512
 
 
 
31
  metrics:
32
  - type: pearson_cosine
33
- value: 0.8509127994264242
34
  name: Pearson Cosine
35
  - type: spearman_cosine
36
- value: 0.8548500966032416
37
  name: Spearman Cosine
38
  - type: pearson_manhattan
39
- value: 0.821303728669975
40
  name: Pearson Manhattan
41
  - type: spearman_manhattan
42
- value: 0.8364598068079891
43
  name: Spearman Manhattan
44
  - type: pearson_euclidean
45
- value: 0.8210450198328316
46
  name: Pearson Euclidean
47
  - type: spearman_euclidean
48
- value: 0.8382181658285147
49
  name: Spearman Euclidean
50
  - type: pearson_dot
51
- value: 0.8491261828772604
52
  name: Pearson Dot
53
  - type: spearman_dot
54
- value: 0.8559811107036664
55
  name: Spearman Dot
56
- - type: pearson_max
57
- value: 0.8509127994264242
58
- name: Pearson Max
59
- - type: spearman_max
60
- value: 0.8559811107036664
61
- name: Spearman Max
62
  - task:
63
  type: semantic-similarity
64
  name: Semantic Similarity
65
  dataset:
66
- name: sts dev 256
67
- type: sts-dev-256
 
 
 
68
  metrics:
69
  - type: pearson_cosine
70
- value: 0.8498025312190702
71
  name: Pearson Cosine
72
  - type: spearman_cosine
73
- value: 0.8530609768738506
74
  name: Spearman Cosine
75
  - type: pearson_manhattan
76
- value: 0.8181745876468085
77
  name: Pearson Manhattan
78
  - type: spearman_manhattan
79
- value: 0.8328727236454085
80
  name: Spearman Manhattan
81
  - type: pearson_euclidean
82
- value: 0.8193792688284338
83
  name: Pearson Euclidean
84
  - type: spearman_euclidean
85
- value: 0.8338632184708783
86
  name: Spearman Euclidean
87
  - type: pearson_dot
88
- value: 0.8396368156921546
89
  name: Pearson Dot
90
  - type: spearman_dot
91
- value: 0.8484397673758116
92
  name: Spearman Dot
93
- - type: pearson_max
94
- value: 0.8498025312190702
95
- name: Pearson Max
96
- - type: spearman_max
97
- value: 0.8530609768738506
98
- name: Spearman Max
99
  license: apache-2.0
100
  language:
101
  - ar
@@ -310,38 +302,19 @@ You can finetune this model on your own dataset.
310
  ### Metrics
311
 
312
  #### Semantic Similarity
313
- * Dataset: `sts-dev-512`
314
  * Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
315
 
316
  | Metric | Value |
317
  |:--------------------|:-----------|
318
- | pearson_cosine | 0.8509 |
319
- | **spearman_cosine** | **0.8549** |
320
- | pearson_manhattan | 0.8213 |
321
- | spearman_manhattan | 0.8365 |
322
- | pearson_euclidean | 0.821 |
323
- | spearman_euclidean | 0.8382 |
324
- | pearson_dot | 0.8491 |
325
- | spearman_dot | 0.856 |
326
- | pearson_max | 0.8509 |
327
- | spearman_max | 0.856 |
328
-
329
- #### Semantic Similarity
330
- * Dataset: `sts-dev-256`
331
- * Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
332
-
333
- | Metric | Value |
334
- |:--------------------|:-----------|
335
- | pearson_cosine | 0.8498 |
336
- | **spearman_cosine** | **0.8531** |
337
- | pearson_manhattan | 0.8182 |
338
- | spearman_manhattan | 0.8329 |
339
- | pearson_euclidean | 0.8194 |
340
- | spearman_euclidean | 0.8339 |
341
- | pearson_dot | 0.8396 |
342
- | spearman_dot | 0.8484 |
343
- | pearson_max | 0.8498 |
344
- | spearman_max | 0.8531 |
345
 
346
  <!--
347
  ## Bias, Risks and Limitations
@@ -402,38 +375,6 @@ Phase `2` produces a finetuned `STS` model based on the model from phase `1`, wi
402
 
403
  </details>
404
 
405
- ### Training Logs (Phase 2)
406
- | Epoch | Step | Training Loss | Validation Loss | sts-dev-512_spearman_cosine | sts-dev-256_spearman_cosine |
407
- |:------:|:----:|:-------------:|:---------------:|:---------------------------:|:---------------------------:|
408
- | 0.3650 | 50 | 0.0395 | 0.0424 | 0.8486 | 0.8487 |
409
- | 0.7299 | 100 | 0.031 | 0.0427 | 0.8493 | 0.8495 |
410
- | 1.0949 | 150 | 0.0344 | 0.0430 | 0.8496 | 0.8496 |
411
- | 1.4599 | 200 | 0.0313 | 0.0427 | 0.8506 | 0.8504 |
412
- | 1.8248 | 250 | 0.0267 | 0.0428 | 0.8504 | 0.8506 |
413
- | 2.1898 | 300 | 0.0309 | 0.0429 | 0.8516 | 0.8515 |
414
- | 2.5547 | 350 | 0.0276 | 0.0425 | 0.8531 | 0.8521 |
415
- | 2.9197 | 400 | 0.028 | 0.0426 | 0.8530 | 0.8515 |
416
- | 3.2847 | 450 | 0.0281 | 0.0425 | 0.8539 | 0.8521 |
417
- | 3.6496 | 500 | 0.0248 | 0.0425 | 0.8542 | 0.8523 |
418
- | 4.0146 | 550 | 0.0302 | 0.0424 | 0.8541 | 0.8520 |
419
- | 4.3796 | 600 | 0.0261 | 0.0421 | 0.8545 | 0.8523 |
420
- | 4.7445 | 650 | 0.0233 | 0.0420 | 0.8544 | 0.8522 |
421
- | 5.1095 | 700 | 0.0281 | 0.0419 | 0.8547 | 0.8528 |
422
- | 5.4745 | 750 | 0.0257 | 0.0419 | 0.8546 | 0.8531 |
423
- | 5.8394 | 800 | 0.0235 | 0.0418 | 0.8546 | 0.8527 |
424
- | 6.2044 | 850 | 0.0268 | 0.0418 | 0.8551 | 0.8529 |
425
- | 6.5693 | 900 | 0.0238 | 0.0416 | 0.8552 | 0.8526 |
426
- | 6.9343 | 950 | 0.0255 | 0.0416 | 0.8549 | 0.8526 |
427
- | 7.2993 | 1000 | 0.0253 | 0.0416 | 0.8548 | 0.8528 |
428
- | 7.6642 | 1050 | 0.0225 | 0.0415 | 0.8550 | 0.8525 |
429
- | 8.0292 | 1100 | 0.0276 | 0.0414 | 0.8550 | 0.8528 |
430
- | 8.3942 | 1150 | 0.0244 | 0.0415 | 0.8550 | 0.8533 |
431
- | 8.7591 | 1200 | 0.0218 | 0.0414 | 0.8551 | 0.8529 |
432
- | 9.1241 | 1250 | 0.0263 | 0.0414 | 0.8550 | 0.8531 |
433
- | 9.4891 | 1300 | 0.0241 | 0.0414 | 0.8552 | 0.8533 |
434
- | 9.8540 | 1350 | 0.0227 | 0.0415 | 0.8549 | 0.8531 |
435
-
436
-
437
  ### Framework Versions
438
  - Python: 3.10.14
439
  - Sentence Transformers: 3.2.0
@@ -443,11 +384,7 @@ Phase `2` produces a finetuned `STS` model based on the model from phase `1`, wi
443
  - Datasets: 3.0.1
444
  - Tokenizers: 0.20.1
445
 
446
- ## Citation
447
-
448
- ### BibTeX
449
-
450
- #### Sentence Transformers
451
  ```bibtex
452
  @inproceedings{reimers-2019-sentence-bert,
453
  title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
 
10
  - spearman_euclidean
11
  - pearson_dot
12
  - spearman_dot
 
 
13
  pipeline_tag: sentence-similarity
14
  tags:
15
  - sentence-transformers
 
24
  type: semantic-similarity
25
  name: Semantic Similarity
26
  dataset:
27
+ config: ar-ar
28
+ name: MTEB STS17 (ar-ar)
29
+ revision: faeb762787bd10488a50c8b5be4a3b82e411949c
30
+ split: test
31
+ type: mteb/sts17-crosslingual-sts
32
  metrics:
33
  - type: pearson_cosine
34
+ value: 0.8515496450525244
35
  name: Pearson Cosine
36
  - type: spearman_cosine
37
+ value: 0.8558624740720275
38
  name: Spearman Cosine
39
  - type: pearson_manhattan
40
+ value: 0.821963706969713
41
  name: Pearson Manhattan
42
  - type: spearman_manhattan
43
+ value: 0.8396900657477299
44
  name: Spearman Manhattan
45
  - type: pearson_euclidean
46
+ value: 0.8231208177674895
47
  name: Pearson Euclidean
48
  - type: spearman_euclidean
49
+ value: 0.8444168331737782
50
  name: Spearman Euclidean
51
  - type: pearson_dot
52
+ value: 0.8515496381581389
53
  name: Pearson Dot
54
  - type: spearman_dot
55
+ value: 0.8557531503465841
56
  name: Spearman Dot
 
 
 
 
 
 
57
  - task:
58
  type: semantic-similarity
59
  name: Semantic Similarity
60
  dataset:
61
+ config: en-ar
62
+ name: MTEB STS17 (en-ar)
63
+ revision: faeb762787bd10488a50c8b5be4a3b82e411949c
64
+ split: test
65
+ type: mteb/sts17-crosslingual-sts
66
  metrics:
67
  - type: pearson_cosine
68
+ value: 0.4960250395119053
69
  name: Pearson Cosine
70
  - type: spearman_cosine
71
+ value: 0.4770240652715316
72
  name: Spearman Cosine
73
  - type: pearson_manhattan
74
+ value: 0.463401831917928
75
  name: Pearson Manhattan
76
  - type: spearman_manhattan
77
+ value: 0.4468968000990917
78
  name: Spearman Manhattan
79
  - type: pearson_euclidean
80
+ value: 0.4481739880481376
81
  name: Pearson Euclidean
82
  - type: spearman_euclidean
83
+ value: 0.428311112429714
84
  name: Spearman Euclidean
85
  - type: pearson_dot
86
+ value: 0.49602504450181617
87
  name: Pearson Dot
88
  - type: spearman_dot
89
+ value: 0.4770240652715316
90
  name: Spearman Dot
 
 
 
 
 
 
91
  license: apache-2.0
92
  language:
93
  - ar
 
302
  ### Metrics
303
 
304
  #### Semantic Similarity
305
+ * Dataset: `MTEB STS17 (ar-ar)` [source](https://huggingface.co/datasets/mteb/sts17-crosslingual-sts/viewer/ar-ar)
306
  * Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)
307
 
308
  | Metric | Value |
309
  |:--------------------|:-----------|
310
+ | pearson_cosine | 0.8515 |
311
+ | **spearman_cosine** | **0.8559** |
312
+ | pearson_manhattan | 0.8220 |
313
+ | spearman_manhattan | 0.8397 |
314
+ | pearson_euclidean | 0.8231 |
315
+ | spearman_euclidean | 0.8444 |
316
+ | pearson_dot | 0.8515 |
317
+ | spearman_dot | 0.8557 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
318
 
319
  <!--
320
  ## Bias, Risks and Limitations
 
375
 
376
  </details>
377
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
378
  ### Framework Versions
379
  - Python: 3.10.14
380
  - Sentence Transformers: 3.2.0
 
384
  - Datasets: 3.0.1
385
  - Tokenizers: 0.20.1
386
 
387
+ #### Sentence Transformers Citation
 
 
 
 
388
  ```bibtex
389
  @inproceedings{reimers-2019-sentence-bert,
390
  title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",