Hugging Face's logo

nlpai-lab
/
KoE5

Feature Extraction
Transformers
Safetensors
Korean
English
xlm-roberta
text-embeddings-inference
Inference Endpoints
Model card Files Community
3
yjoonjang commited on
Commit
bc6d284
Β·
verified Β·
1 Parent(s): 17ffe04

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +110 -11
README.md CHANGED
@@ -14,13 +14,18 @@ pipeline_tag: feature-extraction
14
  # πŸ”Ž KoE5
15
 
16
  Introducing KoE5, a model with advanced retrieval abilities.
17
- It has shown remarkable performance in Korean text retrieval, speficially overwhelming most multilingual embedding models.
18
- To our knowledge, It is one of the best publicly opened Korean retrieval models.
19
 
20
- For details, visit the [KoE5 repository](https://github.com/nlpai-lab/KoE5)
21
 
22
  ---
23
 
 
 
 
 
 
 
24
  ### Model Description
25
 
26
  This is the model card of a πŸ€— transformers model that has been pushed on the Hub.
@@ -81,17 +86,105 @@ print(similarities)
81
 
82
  ## Evaluation
83
  ### Metrics
84
- - NDCG@1, F1@1, NDCG@3, F1@3
85
  ### Benchmark Datasets
86
- - Ko-strategyQA
87
- - AutoRAG-benchmark
88
- - PublicHealthQA
 
 
 
 
 
89
 
90
  ## Results
91
- - By datasets
92
- <img src="KoE5-results-by-datasets.png" width=100%>
93
- - Average
94
- <img src="KoE5-results-avg.png" width=100%>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
95
 
96
  ## FAQ
97
 
@@ -110,6 +203,12 @@ Here are some rules of thumb:
110
 
111
  If you find our paper or models helpful, please consider cite as follows:
112
  ```text
 
 
 
 
 
 
113
  @misc{KoE5,
114
  author = {NLP & AI Lab and Human-Inspired AI research},
115
  title = {KoE5: A New Dataset and Model for Improving Korean Embedding Performance},
 
14
  # πŸ”Ž KoE5
15
 
16
  Introducing KoE5, a model with advanced retrieval abilities.
17
+ It has shown remarkable performance in Korean text retrieval.
 
18
 
19
+ For details, visit the [KURE repository](https://github.com/nlpai-lab/KURE)
20
 
21
  ---
22
 
23
+ ## Model Versions
24
+ | Model Name | Dimension | Sequence Length | Introduction |
25
+ |:----:|:---:|:---:|:---:|
26
+ | [KURE-v1](https://huggingface.co/nlpai-lab/KURE-v1) | 1024 | 8192 | Fine-tuned [BAAI/bge-m3](https://huggingface.co/BAAI/bge-m3) with Korean data via [CachedGISTEmbedLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedgistembedloss)
27
+ | [KoE5](https://huggingface.co/nlpai-lab/KoE5) | 1024 | 512 | Fine-tuned [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) with [ko-triplet-v1.0](https://huggingface.co/datasets/nlpai-lab/ko-triplet-v1.0) via [CachedMultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) |
28
+
29
  ### Model Description
30
 
31
  This is the model card of a πŸ€— transformers model that has been pushed on the Hub.
 
86
 
87
  ## Evaluation
88
  ### Metrics
89
+ - Recall, Precision, NDCG, F1
90
  ### Benchmark Datasets
91
+ - [Ko-StrategyQA](https://huggingface.co/datasets/taeminlee/Ko-StrategyQA): ν•œκ΅­μ–΄ ODQA multi-hop 검색 데이터셋 (StrategyQA λ²ˆμ—­)
92
+ - [AutoRAGRetrieval](https://huggingface.co/datasets/yjoonjang/markers_bm): 금육, 곡곡, 의료, 법λ₯ , 컀머슀 5개 뢄야에 λŒ€ν•΄, pdfλ₯Ό νŒŒμ‹±ν•˜μ—¬ κ΅¬μ„±ν•œ ν•œκ΅­μ–΄ λ¬Έμ„œ 검색 데이터셋
93
+ - [MIRACLRetrieval]([url](https://huggingface.co/datasets/miracl/miracl)): Wikipedia 기반의 ν•œκ΅­μ–΄ λ¬Έμ„œ 검색 데이터셋
94
+ - [PublicHealthQA]([url](https://huggingface.co/datasets/xhluca/publichealth-qa)): 의료 및 곡쀑보건 도메인에 λŒ€ν•œ ν•œκ΅­μ–΄ λ¬Έμ„œ 검색 데이터셋
95
+ - [BelebeleRetrieval]([url](https://huggingface.co/datasets/facebook/belebele)): FLORES-200 기반의 ν•œκ΅­μ–΄ λ¬Έμ„œ 검색 데이터셋
96
+ - [MrTidyRetrieval](https://huggingface.co/datasets/mteb/mrtidy): Wikipedia 기반의 ν•œκ΅­μ–΄ λ¬Έμ„œ 검색 데이터셋
97
+ - [MultiLongDocRetrieval](https://huggingface.co/datasets/Shitao/MLDR): λ‹€μ–‘ν•œ λ„λ©”μΈμ˜ ν•œκ΅­μ–΄ μž₯λ¬Έ 검색 데이터셋
98
+ - [XPQARetrieval](https://huggingface.co/datasets/jinaai/xpqa): λ‹€μ–‘ν•œ λ„λ©”μΈμ˜ ν•œκ΅­μ–΄ λ¬Έμ„œ 검색 데이터셋
99
 
100
  ## Results
101
+
102
+ μ•„λž˜λŠ” λͺ¨λ“  λͺ¨λΈμ˜, λͺ¨λ“  벀치마크 데이터셋에 λŒ€ν•œ 평균 κ²°κ³Όμž…λ‹ˆλ‹€.
103
+ μžμ„Έν•œ κ²°κ³ΌλŠ” [KURE Github](https://github.com/nlpai-lab/KURE/tree/main/eval/results)μ—μ„œ ν™•μΈν•˜μ‹€ 수 μžˆμŠ΅λ‹ˆλ‹€.
104
+ ### Top-k 1
105
+ | Model | Average Recall_top1 | Average Precision_top1 | Average NDCG_top1 | Average F1_top1 |
106
+ |-----------------------------------------|----------------------|------------------------|-------------------|-----------------|
107
+ | **nlpai-lab/KURE-v1** | **0.52640** | **0.60551** | **0.60551** | **0.55784** |
108
+ | dragonkue/BGE-m3-ko | 0.52361 | 0.60394 | 0.60394 | 0.55535 |
109
+ | BAAI/bge-m3 | 0.51778 | 0.59846 | 0.59846 | 0.54998 |
110
+ | Snowflake/snowflake-arctic-embed-l-v2.0 | 0.51246 | 0.59384 | 0.59384 | 0.54489 |
111
+ | nlpai-lab/KoE5 | 0.50157 | 0.57790 | 0.57790 | 0.53178 |
112
+ | intfloat/multilingual-e5-large | 0.50052 | 0.57727 | 0.57727 | 0.53122 |
113
+ | jinaai/jina-embeddings-v3 | 0.48287 | 0.56068 | 0.56068 | 0.51361 |
114
+ | BAAI/bge-multilingual-gemma2 | 0.47904 | 0.55472 | 0.55472 | 0.50916 |
115
+ | intfloat/multilingual-e5-large-instruct | 0.47842 | 0.55435 | 0.55435 | 0.50826 |
116
+ | intfloat/multilingual-e5-base | 0.46950 | 0.54490 | 0.54490 | 0.49947 |
117
+ | intfloat/e5-mistral-7b-instruct | 0.46772 | 0.54394 | 0.54394 | 0.49781 |
118
+ | Alibaba-NLP/gte-multilingual-base | 0.46469 | 0.53744 | 0.53744 | 0.49353 |
119
+ | Alibaba-NLP/gte-Qwen2-7B-instruct | 0.46633 | 0.53625 | 0.53625 | 0.49429 |
120
+ | openai/text-embedding-3-large | 0.44884 | 0.51688 | 0.51688 | 0.47572 |
121
+ | Salesforce/SFR-Embedding-2_R | 0.43748 | 0.50815 | 0.50815 | 0.46504 |
122
+ | upskyy/bge-m3-korean | 0.43125 | 0.50245 | 0.50245 | 0.45945 |
123
+ | jhgan/ko-sroberta-multitask | 0.33788 | 0.38497 | 0.38497 | 0.35678 |
124
+
125
+ ### Top-k 3
126
+ | Model | Average Recall_top1 | Average Precision_top1 | Average NDCG_top1 | Average F1_top1 |
127
+ |-----------------------------------------|----------------------|------------------------|-------------------|-----------------|
128
+ | **nlpai-lab/KURE-v1** | **0.68678** | **0.28711** | **0.65538** | **0.39835** |
129
+ | dragonkue/BGE-m3-ko | 0.67834 | 0.28385 | 0.64950 | 0.39378 |
130
+ | BAAI/bge-m3 | 0.67526 | 0.28374 | 0.64556 | 0.39291 |
131
+ | Snowflake/snowflake-arctic-embed-l-v2.0 | 0.67128 | 0.28193 | 0.64042 | 0.39072 |
132
+ | intfloat/multilingual-e5-large | 0.65807 | 0.27777 | 0.62822 | 0.38423 |
133
+ | nlpai-lab/KoE5 | 0.65174 | 0.27329 | 0.62369 | 0.37882 |
134
+ | BAAI/bge-multilingual-gemma2 | 0.64415 | 0.27416 | 0.61105 | 0.37782 |
135
+ | jinaai/jina-embeddings-v3 | 0.64116 | 0.27165 | 0.60954 | 0.37511 |
136
+ | intfloat/multilingual-e5-large-instruct | 0.64353 | 0.27040 | 0.60790 | 0.37453 |
137
+ | Alibaba-NLP/gte-multilingual-base | 0.63744 | 0.26404 | 0.59695 | 0.36764 |
138
+ | Alibaba-NLP/gte-Qwen2-7B-instruct | 0.63163 | 0.25937 | 0.59237 | 0.36263 |
139
+ | intfloat/multilingual-e5-base | 0.62099 | 0.26144 | 0.59179 | 0.36203 |
140
+ | intfloat/e5-mistral-7b-instruct | 0.62087 | 0.26144 | 0.58917 | 0.36188 |
141
+ | openai/text-embedding-3-large | 0.61035 | 0.25356 | 0.57329 | 0.35270 |
142
+ | Salesforce/SFR-Embedding-2_R | 0.60001 | 0.25253 | 0.56346 | 0.34952 |
143
+ | upskyy/bge-m3-korean | 0.59215 | 0.25076 | 0.55722 | 0.34623 |
144
+ | jhgan/ko-sroberta-multitask | 0.46930 | 0.18994 | 0.43293 | 0.26696 |
145
+
146
+ ### Top-k 5
147
+ | Model | Average Recall_top1 | Average Precision_top1 | Average NDCG_top1 | Average F1_top1 |
148
+ |-----------------------------------------|----------------------|------------------------|-------------------|-----------------|
149
+ | **nlpai-lab/KURE-v1** | **0.73851** | **0.19130** | **0.67479** | **0.29903** |
150
+ | dragonkue/BGE-m3-ko | 0.72517 | 0.18799 | 0.66692 | 0.29401 |
151
+ | BAAI/bge-m3 | 0.72954 | 0.18975 | 0.66615 | 0.29632 |
152
+ | Snowflake/snowflake-arctic-embed-l-v2.0 | 0.72962 | 0.18875 | 0.66236 | 0.29542 |
153
+ | nlpai-lab/KoE5 | 0.70820 | 0.18287 | 0.64499 | 0.28628 |
154
+ | intfloat/multilingual-e5-large | 0.70124 | 0.18316 | 0.64402 | 0.28588 |
155
+ | BAAI/bge-multilingual-gemma2 | 0.70258 | 0.18556 | 0.63338 | 0.28851 |
156
+ | jinaai/jina-embeddings-v3 | 0.69933 | 0.18256 | 0.63133 | 0.28505 |
157
+ | intfloat/multilingual-e5-large-instruct | 0.69018 | 0.17838 | 0.62486 | 0.27933 |
158
+ | Alibaba-NLP/gte-multilingual-base | 0.69365 | 0.17789 | 0.61896 | 0.27879 |
159
+ | intfloat/multilingual-e5-base | 0.67250 | 0.17406 | 0.61119 | 0.27247 |
160
+ | Alibaba-NLP/gte-Qwen2-7B-instruct | 0.67447 | 0.17114 | 0.60952 | 0.26943 |
161
+ | intfloat/e5-mistral-7b-instruct | 0.67449 | 0.17484 | 0.60935 | 0.27349 |
162
+ | openai/text-embedding-3-large | 0.66365 | 0.17004 | 0.59389 | 0.26677 |
163
+ | Salesforce/SFR-Embedding-2_R | 0.65622 | 0.17018 | 0.58494 | 0.26612 |
164
+ | upskyy/bge-m3-korean | 0.65477 | 0.17015 | 0.58073 | 0.26589 |
165
+ | jhgan/ko-sroberta-multitask | 0.53136 | 0.13264 | 0.45879 | 0.20976 |
166
+
167
+ ### Top-k 10
168
+ | Model | Average Recall_top1 | Average Precision_top1 | Average NDCG_top1 | Average F1_top1 |
169
+ |-----------------------------------------|----------------------|------------------------|-------------------|-----------------|
170
+ | **nlpai-lab/KURE-v1** | **0.79682** | **0.10624** | **0.69473** | **0.18524** |
171
+ | dragonkue/BGE-m3-ko | 0.78450 | 0.10492 | 0.68748 | 0.18288 |
172
+ | BAAI/bge-m3 | 0.79195 | 0.10592 | 0.68723 | 0.18456 |
173
+ | Snowflake/snowflake-arctic-embed-l-v2.0 | 0.78669 | 0.10462 | 0.68189 | 0.18260 |
174
+ | intfloat/multilingual-e5-large | 0.75902 | 0.10147 | 0.66370 | 0.17693 |
175
+ | nlpai-lab/KoE5 | 0.75296 | 0.09937 | 0.66012 | 0.17369 |
176
+ | BAAI/bge-multilingual-gemma2 | 0.76153 | 0.10364 | 0.65330 | 0.18003 |
177
+ | jinaai/jina-embeddings-v3 | 0.76277 | 0.10240 | 0.65290 | 0.17843 |
178
+ | intfloat/multilingual-e5-large-instruct | 0.74851 | 0.09888 | 0.64451 | 0.17283 |
179
+ | Alibaba-NLP/gte-multilingual-base | 0.75631 | 0.09938 | 0.64025 | 0.17363 |
180
+ | Alibaba-NLP/gte-Qwen2-7B-instruct | 0.74092 | 0.09607 | 0.63258 | 0.16847 |
181
+ | intfloat/multilingual-e5-base | 0.73512 | 0.09717 | 0.63216 | 0.16977 |
182
+ | intfloat/e5-mistral-7b-instruct | 0.73795 | 0.09777 | 0.63076 | 0.17078 |
183
+ | openai/text-embedding-3-large | 0.72946 | 0.09571 | 0.61670 | 0.16739 |
184
+ | Salesforce/SFR-Embedding-2_R | 0.71662 | 0.09546 | 0.60589 | 0.16651 |
185
+ | upskyy/bge-m3-korean | 0.71895 | 0.09583 | 0.60258 | 0.16712 |
186
+ | jhgan/ko-sroberta-multitask | 0.61225 | 0.07826 | 0.48687 | 0.13757 |
187
+ <br/>
188
 
189
  ## FAQ
190
 
 
203
 
204
  If you find our paper or models helpful, please consider cite as follows:
205
  ```text
206
+ @misc{KURE,
207
+ publisher = {Youngjoon Jang, Junyoung Son, Taemin Lee},
208
+ year = {2024},
209
+ url = {https://github.com/nlpai-lab/KURE}
210
+ },
211
+
212
  @misc{KoE5,
213
  author = {NLP & AI Lab and Human-Inspired AI research},
214
  title = {KoE5: A New Dataset and Model for Improving Korean Embedding Performance},