first model version

1_Pooling/config.json

@@ -0,0 +1,7 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false
+}

README.md

@@ -0,0 +1,126 @@
+---
+pipeline_tag: sentence-similarity
+tags:
+- sentence-transformers
+- feature-extraction
+- sentence-similarity
+- transformers
+
+---
+
+# {MODEL_NAME}
+
+This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
+
+<!--- Describe your model here -->
+
+## Usage (Sentence-Transformers)
+
+Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+
+```
+pip install -U sentence-transformers
+```
+
+Then you can use the model like this:
+
+```python
+from sentence_transformers import SentenceTransformer
+sentences = ["This is an example sentence", "Each sentence is converted"]
+
+model = SentenceTransformer('{MODEL_NAME}')
+embeddings = model.encode(sentences)
+print(embeddings)
+```
+
+
+
+## Usage (HuggingFace Transformers)
+Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
+
+```python
+from transformers import AutoTokenizer, AutoModel
+import torch
+
+
+#Mean Pooling - Take attention mask into account for correct averaging
+def mean_pooling(model_output, attention_mask):
+    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
+    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+
+
+# Sentences we want sentence embeddings for
+sentences = ['This is an example sentence', 'Each sentence is converted']
+
+# Load model from HuggingFace Hub
+tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
+model = AutoModel.from_pretrained('{MODEL_NAME}')
+
+# Tokenize sentences
+encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+
+# Compute token embeddings
+with torch.no_grad():
+    model_output = model(**encoded_input)
+
+# Perform pooling. In this case, mean pooling.
+sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
+
+print("Sentence embeddings:")
+print(sentence_embeddings)
+```
+
+
+
+## Evaluation Results
+
+<!--- Describe how your model was evaluated -->
+
+For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})
+
+
+## Training
+The model was trained with the parameters:
+
+**DataLoader**:
+
+`torch.utils.data.dataloader.DataLoader` of length 180 with parameters:
+```
+{'batch_size': 32, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
+```
+
+**Loss**:
+
+`sentence_transformers.losses.CosineSimilarityLoss.CosineSimilarityLoss` 
+
+Parameters of the fit()-Method:
+```
+{
+    "epochs": 5,
+    "evaluation_steps": 25,
+    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
+    "max_grad_norm": 1,
+    "optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
+    "optimizer_params": {
+        "lr": 2e-05
+    },
+    "scheduler": "WarmupLinear",
+    "steps_per_epoch": null,
+    "warmup_steps": 90,
+    "weight_decay": 0.01
+}
+```
+
+
+## Full Model Architecture
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: MPNetModel 
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
+)
+```
+
+## Citing & Authors
+
+<!--- Describe where people can find more information -->

config.json

@@ -0,0 +1,24 @@
+{
+  "_name_or_path": "/home/jupyter/.cache/torch/sentence_transformers/jamescalam_mpnet-snli/",
+  "architectures": [
+    "MPNetModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "mpnet",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "relative_attention_num_buckets": 32,
+  "torch_dtype": "float32",
+  "transformers_version": "4.21.3",
+  "vocab_size": 30527
+}

config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.2.2",
+    "transformers": "4.21.3",
+    "pytorch": "1.12.1.post200"
+  }
+}

eval/similarity_evaluation_results.csv

@@ -0,0 +1,41 @@
+epoch,steps,cosine_pearson,cosine_spearman,euclidean_pearson,euclidean_spearman,manhattan_pearson,manhattan_spearman,dot_pearson,dot_spearman
+0,25,0.8525281011730752,0.8564233177667847,0.8555224104298315,0.8526796997850682,0.8557031160381019,0.8530282338331145,0.8313752495650653,0.8269044603353088
+0,50,0.8741100140610053,0.8727017606237887,0.8650925639108137,0.8658785327254077,0.8659407105012707,0.8666661560440556,0.8550144317932907,0.8488134818178953
+0,75,0.880021536875125,0.8783739335773616,0.8689498177036719,0.8717629656067097,0.8709873088316269,0.8737577890456425,0.8623350931749131,0.8579931401529279
+0,100,0.8903371178412601,0.8898073801785448,0.879586482659841,0.8842451297999553,0.8804405955422622,0.8847968785045572,0.8681473020378078,0.8647414882264052
+0,125,0.8923719246946223,0.8921381600613835,0.8844429068888086,0.8871154111401977,0.885376254454852,0.8878247240077843,0.869681486585762,0.8654224335361903
+0,150,0.8929799221840768,0.8925860529121719,0.884470121951992,0.8878583112195456,0.8856201618700077,0.8886821641005301,0.873595985155225,0.8699944242358539
+0,175,0.8918725777110812,0.8918301635042188,0.8845454724017465,0.8871811577279256,0.8858635448826278,0.8881875997027262,0.8729038114238787,0.8687024314688533
+0,-1,0.8927419380983888,0.8930545015377597,0.8857048952861747,0.8882851976347614,0.8868979491028508,0.8894342143418521,0.8738962937256752,0.8699049305182823
+1,25,0.8938486519455097,0.8937914175267404,0.8867878401759101,0.8886953829708508,0.8877554627829624,0.8895487536657741,0.8754243024729874,0.8712594945039102
+1,50,0.8936252750558363,0.8940830351657113,0.8878685943987911,0.8898342431863832,0.8883154852333333,0.8900131739146883,0.8763777990900758,0.872582319260222
+1,75,0.8980799807291938,0.8980766864068954,0.8904505924192763,0.8936865198628083,0.8904906292072985,0.8934669111705241,0.8789415974823566,0.8751584206821453
+1,100,0.8971584397667315,0.8965754710494689,0.889670996081187,0.8924558321818515,0.8902884662180718,0.8926442700341469,0.8784868397927074,0.8740551475660299
+1,125,0.8974913661405046,0.8979483510004668,0.8907384377271917,0.8930089855880079,0.8910214035999827,0.8930091813176801,0.8796114235331997,0.8755660811058249
+1,150,0.8979507357653558,0.8983208548162545,0.8910434024689661,0.8943787036138979,0.891339847948966,0.8943903815578897,0.8778382316965467,0.8746485306505495
+1,175,0.8972791906700768,0.8979103394083983,0.8907270543540502,0.8943680915069397,0.8900666213703706,0.893247260567582,0.8776959875985283,0.8742579431923888
+1,-1,0.8976134926112977,0.8981373173230452,0.8906819209264969,0.8945833745734669,0.8900502758501823,0.8935425678062981,0.8779817808480646,0.874641950574834
+2,25,0.8969885361631248,0.8968703111105923,0.8901574634631075,0.8931725879181848,0.8909172875170468,0.8941172692700107,0.8765027880073822,0.8721675396145476
+2,50,0.8984530749883232,0.8983518886455005,0.8918926468783053,0.8949179246261102,0.8923588047118551,0.8950830848822909,0.8783821413429027,0.8739055720656471
+2,75,0.8991549148499863,0.8986274938179051,0.8912345734582248,0.8944484563310122,0.8913466895128087,0.8943684776282023,0.8762440886363715,0.8718303276380618
+2,100,0.8996257715500752,0.8991293233457175,0.8910857354953711,0.8938026533948076,0.8913129712238838,0.893839703242407,0.8768981612610761,0.8723712176910511
+2,125,0.898079081352657,0.8983873637589502,0.8917669473330797,0.8944306688575743,0.8918436152814067,0.8942238013719045,0.8772510512994548,0.8725326252757497
+2,150,0.8994910810023666,0.900046348089557,0.8921071090774162,0.8959934947624345,0.8925312881929448,0.8960656958798127,0.8816324221556006,0.8782754388474677
+2,175,0.900460406431834,0.900813395682291,0.8938064043404844,0.8978592736284409,0.894124968934986,0.8977953625523637,0.882135740398583,0.8787344658546387
+2,-1,0.9007342201794905,0.9010681414096952,0.8938665891125965,0.8979887879525805,0.8941257479715393,0.89790769714936,0.8822329105730269,0.8788371491995228
+3,25,0.9001397925435157,0.9006539400562349,0.8932753292924296,0.8965231410349515,0.8937971409233019,0.8967794312471794,0.8811540148483646,0.8771494896718617
+3,50,0.8995918856471997,0.8996281635476238,0.8929552997939237,0.8960779094113621,0.8935836610462061,0.896385813538167,0.879936977211149,0.8755099956578671
+3,75,0.8992108631274484,0.899581916184747,0.8927565092810681,0.8960267456750313,0.893258811079497,0.8961767030737589,0.8792891031487517,0.8749802177121905
+3,100,0.8998207692174952,0.9002932565333019,0.8937293776642574,0.8971820454036734,0.8939766089899452,0.8968554473138062,0.88000327136521,0.8759460475602144
+3,125,0.899059165031997,0.8995598592300236,0.8933086851077897,0.8962789904178182,0.8937500679131108,0.8963536106689971,0.8800787902255429,0.8760055031172599
+3,150,0.899829195296883,0.9002656248416351,0.8933329201520631,0.8967788031328675,0.893616787429137,0.8965955325439199,0.8809039962501821,0.8768603008601897
+3,175,0.9001110099345248,0.9002200020343496,0.892830423421675,0.8967290174187781,0.8930373053969737,0.8965043527657744,0.8791530491966453,0.8750244099135065
+3,-1,0.9002924729109661,0.900298712053081,0.8929868881725396,0.8969162445087339,0.8931762982018293,0.896725801316978,0.8790882148000395,0.8748309783974076
+4,25,0.9004553283790706,0.900561854583433,0.8936912086340808,0.897026620029647,0.8939368087167784,0.8968348067301853,0.8790824711174734,0.8745392156284908
+4,50,0.9005184727159057,0.9007347283682154,0.893989687052925,0.8971623549986433,0.8943230658237618,0.8970158574743807,0.8796850210972333,0.8752169625374921
+4,75,0.9003609882105141,0.9004004897033899,0.8935409368933861,0.8968941982301938,0.8940225986384088,0.8968550113704453,0.8796829915332844,0.8751470978323659
+4,100,0.9000997223272633,0.9003048241569431,0.8936108418808617,0.8969843940218961,0.8941265379685396,0.8970786289622898,0.8801705164595555,0.8759594115594901
+4,125,0.8999246666827019,0.9001625874039797,0.8935260241791758,0.8968941946714727,0.8939878113950323,0.8969387844674552,0.8799248863093416,0.8757262174485895
+4,150,0.8999722110125805,0.900328473639456,0.8935627777761065,0.8969962872685274,0.89398204718539,0.896915061233897,0.8798650863671331,0.8756430028670945
+4,175,0.9000850585235145,0.9004041943322812,0.8936486695777016,0.8970291012557625,0.8940636887312164,0.8970128770452777,0.879891704639246,0.8756492608785305
+4,-1,0.9000891936138207,0.9004027103454918,0.8936564219008413,0.8970390310702293,0.8940696360117925,0.8970235862197268,0.8798970913218596,0.8756512608799109

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f8ebfbd72d4f29040e1204d03c1a8f27a109ded4a8760dfe03bc17b03153d50
+size 438014769

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,15 @@
+{
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "unk_token": "[UNK]"
+}

tokenizer_config.json

@@ -0,0 +1,16 @@
+{
+  "bos_token": "<s>",
+  "cls_token": "<s>",
+  "do_lower_case": true,
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "model_max_length": 512,
+  "name_or_path": "/home/jupyter/.cache/torch/sentence_transformers/jamescalam_mpnet-snli/",
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "special_tokens_map_file": null,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "MPNetTokenizer",
+  "unk_token": "[UNK]"
+}