|
--- |
|
pipeline_tag: sentence-similarity |
|
tags: |
|
- sentence-transformers |
|
- feature-extraction |
|
- sentence-similarity |
|
- mteb |
|
- arctic |
|
- snowflake-arctic-embed |
|
- transformers.js |
|
license: apache-2.0 |
|
language: |
|
- af |
|
- ar |
|
- az |
|
- be |
|
- bg |
|
- bn |
|
- ca |
|
- ceb |
|
- cs |
|
- cy |
|
- da |
|
- de |
|
- el |
|
- en |
|
- es |
|
- et |
|
- eu |
|
- fa |
|
- fi |
|
- fr |
|
- gl |
|
- gu |
|
- he |
|
- hi |
|
- hr |
|
- ht |
|
- hu |
|
- hy |
|
- id |
|
- is |
|
- it |
|
- ja |
|
- jv |
|
- ka |
|
- kk |
|
- km |
|
- kn |
|
- ko |
|
- ky |
|
- lo |
|
- lt |
|
- lv |
|
- mk |
|
- ml |
|
- mn |
|
- mr |
|
- ms |
|
- my |
|
- ne |
|
- nl |
|
- 'no' |
|
- pa |
|
- pl |
|
- pt |
|
- qu |
|
- ro |
|
- ru |
|
- si |
|
- sk |
|
- sl |
|
- so |
|
- sq |
|
- sr |
|
- sv |
|
- sw |
|
- ta |
|
- te |
|
- th |
|
- tl |
|
- tr |
|
- uk |
|
- ur |
|
- vi |
|
- yo |
|
- zh |
|
--- |
|
<h1 align="center">Snowflake's Arctic-embed-l-v2.0</h1> |
|
<h4 align="center"> |
|
<p> |
|
<a href=#models>Models</a> | |
|
<a href=#usage>Usage</a> | |
|
<a href="#evaluation">Evaluation</a> | |
|
<a href="#contact">Contact</a> | |
|
<a href="#faq">FAQ</a> |
|
<a href="#license">License</a> | |
|
<a href="#acknowledgement">Acknowledgement</a> |
|
<p> |
|
</h4> |
|
|
|
|
|
## Models |
|
|
|
|
|
MIRACL (4) Voyage misc. (9) CLEF (5) CLEF, max context length Multilingual CLEF |
|
Snowflake's snowflake-arctic-embed-l-v2.0 is a multilingual text embedding models that focuses on providing |
|
BEIR |
|
0.556 0.558 0.655 0.529 0.541 0.543 |
|
0.543 0.543 0.644 0.519 0.528 0.534 |
|
|
|
Focused on |
|
|
|
| Model Name | # params | # non-emb params | # dimensions | BEIR (15) | MIRACL (4) | CLEF (Focused) | CLEF (Full) | |
|
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:| |
|
| me5 base | 560M | 303M | 1024 | 0.514 | 0.540 | 0.430 | 0.346 | |
|
| bge-m3 (BAAI) | 568M | 303M | 1024 | 0.488 | 0.568 | 0.408 | 0.413 | |
|
| gte (Alibaba) | 305M | 113M | 768 | 0.511 | 0.523 | 0.477 | 0.531 | |
|
| Arctic-M | 109M | 86M | 768 | 0.549 | 0.249 | 0.344 | 0.291 | |
|
| snowflake-arctic-m | 335M | 303M | 1024 | 0.560 | 0.348 | 0.382 | 0.337 | |
|
| me5 base | 560M | 303M | 1024 | 0.514 | 0.540 | 0.430 | 0.346 | |
|
| bge-m3 (BAAI) | 568M | 303M | 1024 | 0.488 | 0.568 | 0.408 | 0.413 | |
|
| gte (Alibaba) | 305M | 113M | 768 | 0.511 | 0.523 | 0.477 | 0.531 | |
|
| snowflake-arctic-m | 109M | 86M | 768 | 0.549 | 0.249 | 0.344 | 0.291 | |
|
| snowflake-arctic-l | 335M | 303M | 1024 | 0.560 | 0.348 | 0.382 | 0.337 | |
|
| snowflake-arctic-m-v2.0 | 305M | 113M | 768 | 0.554 | 0.552 | 0.517 | 0.539 | |
|
| snowflake-arctic-l-v2.0 | 568M | 303M | 1024 | 0.556 | 0.558 | 0.529 | 0.543 | |
|
|
|
MRL |
|
|
|
| Model | | BEIR (15) | Relative Performance | MIRACL (4) | Relative Performance | CLEF (5) | Relative Performance | CLEF (Full) | Relative Performance | |
|
|---|---|:---:|:---:|:---:|:---:|:---:|---|---|---| |
|
| snowflake-arctic-l-v2.0 | 1024 | 0.556 | N/A | 0.558 | N/A | 0.529 | N/A | 0.543 | N/A | |
|
| snowflake-arctic-l-v2.0 | 256 | 0.543 | -0.18% | 0.543 | -2.70% | 0.519 | -1.81% | 0.534 | -1.53% | |
|
| snowflake-arctic-m-v2.0 | 768 | 0.554 | N/A | 0.552 | N/A | 0.517 | N/A | 0.539 | N/A | |
|
| snowflake-arctic-m-v2.0 | 256 | 0.544 | -1.81% | 0.54 | -2.17% | 0.506 | -2.13% | 0.523 | -3.06% | |
|
|
|
|
|
|
|
The `snowflake-arctic-embedding` models achieve **state-of-the-art performance on the MTEB/BEIR leaderboard** for each of their size variants. Evaluation is performed using these [scripts](https://github.com/Snowflake-Labs/snowflake-arctic-embed/tree/main/src). As shown below, each class of model size achieves SOTA retrieval accuracy compared to other top models. |
|
|
|
|
|
The models are trained by leveraging existing open-source text representation models, such as bert-base-uncased, and are trained in a multi-stage pipeline to optimize their retrieval performance. First, the models are trained with large batches of query-document pairs where negatives are derived in-batch—pretraining leverages about 400m samples of a mix of public datasets and proprietary web search data. Following pretraining models are further optimized with long training on a smaller dataset (about 1m samples) of triplets of query, positive document, and negative document derived from hard harmful mining. Mining of the negatives and data curation is crucial to retrieval accuracy. A detailed technical report can be found [here](https://arxiv.org/abs/2405.05374). |
|
|
|
|
|
| Name | MTEB Retrieval Score (NDCG @ 10) | Parameters (Millions) | Embedding Dimension | |
|
| ----------------------------------------------------------------------- | -------------------------------- | --------------------- | ------------------- | |
|
| [snowflake-arctic-embed-xs](https://huggingface.co/Snowflake/snowflake-arctic-embed-xs/) | 50.15 | 22 | 384 | |
|
| [snowflake-arctic-embed-s](https://huggingface.co/Snowflake/snowflake-arctic-embed-s/) | 51.98 | 33 | 384 | |
|
| [snowflake-arctic-embed-m](https://huggingface.co/Snowflake/snowflake-arctic-embed-m/) | 54.90 | 110 | 768 | |
|
| [snowflake-arctic-embed-m-long](https://huggingface.co/Snowflake/snowflake-arctic-embed-m-long/) | 54.83 | 137 | 768 | |
|
| [snowflake-arctic-embed-l](https://huggingface.co/Snowflake/snowflake-arctic-embed-l/) | 55.98 | 335 | 1024 | |
|
|
|
|
|
|
|
## Usage |
|
|
|
### Using Huggingface transformers |
|
|
|
|
|
You can use the transformers package to use an snowflake-arctic-embed model, as shown below. For optimal retrieval quality, use the CLS token to embed each text portion and use the query prefix below (just on the query). |
|
|
|
```python |
|
import torch |
|
from transformers import AutoModel, AutoTokenizer |
|
|
|
model_name = 'snowflake-arctic-embed-l-v2.0' |
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
model = AutoModel.from_pretrained(model_name, add_pooling_layer=False) |
|
model.eval() |
|
|
|
query_prefix = 'Represent this sentence for searching relevant passages: ' |
|
queries = ['what is snowflake?', 'Where can I get the best tacos?'] |
|
queries_with_prefix = ["{}{}".format(query_prefix, i) for i in queries] |
|
query_tokens = tokenizer(queries_with_prefix, padding=True, truncation=True, return_tensors='pt', max_length=512) |
|
|
|
documents = ['The Data Cloud!', 'Mexico City of Course!'] |
|
document_tokens = tokenizer(documents, padding=True, truncation=True, return_tensors='pt', max_length=512) |
|
|
|
# Compute token embeddings |
|
with torch.no_grad(): |
|
query_embeddings = model(**query_tokens)[0][:, 0] |
|
document_embeddings = model(**document_tokens)[0][:, 0] |
|
|
|
|
|
# normalize embeddings |
|
query_embeddings = torch.nn.functional.normalize(query_embeddings, p=2, dim=1) |
|
document_embeddings = torch.nn.functional.normalize(document_embeddings, p=2, dim=1) |
|
|
|
scores = torch.mm(query_embeddings, document_embeddings.transpose(0, 1)) |
|
for query, query_scores in zip(queries, scores): |
|
doc_score_pairs = list(zip(documents, query_scores)) |
|
doc_score_pairs = sorted(doc_score_pairs, key=lambda x: x[1], reverse=True) |
|
#Output passages & scores |
|
print("Query:", query) |
|
for document, score in doc_score_pairs: |
|
print(score, document) |
|
``` |
|
|
|
## Contact |
|
|
|
|
|
Feel free to open an issue or pull request if you have any questions or suggestions about this project. |
|
You also can email Daniel Campos([email protected]). |
|
|
|
|
|
## License |
|
Arctic is licensed under the [Apache-2](https://www.apache.org/licenses/LICENSE-2.0). The released models can be used for commercial purposes free of charge. |
