Update README.md
Browse files
README.md
CHANGED
|
@@ -15,20 +15,28 @@ We fine-tuned the [Open-Orca/OpenOrca-Platypus2-13B](https://huggingface.co/Open
|
|
| 15 |
Its performance is competitive, rivaling previous state-of-the-art algorithms and LLMs such as OpenAI's GPT 3.5 and GPT 4 ([as demonstrated in our earlier studies](https://arxiv.org/abs/2308.16361)).
|
| 16 |
It is notable that, as a 13B model, Jellyfish allows for cost-effective local execution without compromising data security.
|
| 17 |
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
|
| 27 |
-
|
|
| 28 |
-
|
|
| 29 |
-
|
|
| 30 |
-
|
|
| 31 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 32 |
|
| 33 |
_Accuracy as the metric for data imputation and the F1 score for other tasks._
|
| 34 |
_For GPT-3.5, GPT-4 we used the few-shot approach, while for Jellyfish and Jellyfish-Reasoning, the zero-shot approach was employed._
|
|
@@ -39,15 +47,8 @@ _For GPT-3.5, GPT-4 we used the few-shot approach, while for Jellyfish and Jelly
|
|
| 39 |
[HoloClean](https://arxiv.org/abs/1702.00820) for Data Imputation
|
| 40 |
2.
|
| 41 |
[Large Language Models as Data Preprocessors](https://arxiv.org/abs/2308.16361)
|
| 42 |
-
|
| 43 |
|
| 44 |
-
We release two distinct versions of Jellyfish: Jellyfish-13B (the main branch) and Jellyfish-13B-Reasoning.
|
| 45 |
-
As the names suggest, Jellyfish-13B is tailored to deliver precise, straightforward answers.
|
| 46 |
-
In contrast, Jellyfish-13B-Reasoning, is fine-tuned with data that includes reasoning and sequential thought processes for handling data preprocessing tasks, distilling knowledge from GPT-4.
|
| 47 |
-
|
| 48 |
-
The two versions are designed for different application scenarios.
|
| 49 |
-
Jellyfish-13B is suitable for integration into larger data management systems due to its simple and clear responses that can be easily transformed into code.
|
| 50 |
-
On the other hand, Jellyfish-13B-Reasoning is more user-oriented, with responses that provide them with in-depth data insights without the necessity for advanced coding skills or an intricate grasp of statistics.
|
| 51 |
|
| 52 |
**Jellyfish paper will be coming soon!**
|
| 53 |
|
|
|
|
| 15 |
Its performance is competitive, rivaling previous state-of-the-art algorithms and LLMs such as OpenAI's GPT 3.5 and GPT 4 ([as demonstrated in our earlier studies](https://arxiv.org/abs/2308.16361)).
|
| 16 |
It is notable that, as a 13B model, Jellyfish allows for cost-effective local execution without compromising data security.
|
| 17 |
|
| 18 |
+
We release two distinct versions of Jellyfish: Jellyfish-13B (the main branch) and Jellyfish-13B-Reasoning.
|
| 19 |
+
As the names suggest, Jellyfish-13B is tailored to deliver precise, straightforward answers.
|
| 20 |
+
In contrast, Jellyfish-13B-Reasoning, is fine-tuned with data that includes reasoning and sequential thought processes for handling data preprocessing tasks, distilling knowledge from GPT-4.
|
| 21 |
+
|
| 22 |
+
The two versions are designed for different application scenarios.
|
| 23 |
+
Jellyfish-13B is suitable for integration into larger data management systems due to its simple and clear responses that can be easily transformed into code.
|
| 24 |
+
On the other hand, Jellyfish-13B-Reasoning is more user-oriented, with responses that provide them with in-depth data insights without the necessity for advanced coding skills or an intricate grasp of statistics.
|
| 25 |
+
|
| 26 |
+
| Task | Dataset | Non-LLM SoTA<sup>1</sup> | GPT-3.5<sup>2</sup> | GPT-4<sup>2</sup> | Jellyfish-13B| Jellyfish-13B-Resoning | Jellyfish-13B-1.1<sup>3</sup> |
|
| 27 |
+
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
|
| 28 |
+
| Entity Matching | Fodors-Zagats | 100 | 100 | 100 | 100 | 100 | 100 |
|
| 29 |
+
| Entity Matching | Beer | 94.37| 96.30 | 100 | 93.33 | 100 | 96.55 |
|
| 30 |
+
| Entity Matching | iTunes-Amazon | 97.06| 96.43 | 100 | 96.30 | 96.15 | 100 |
|
| 31 |
+
| Entity Matching | Walmart-Amazon | 86.76| 86.17 | 90.27 | 80.71 | 85.16 | 89.18 |
|
| 32 |
+
| Entity Matching | DBLP-ACM | 98.99| 96.99 | 97.44 | 97.35 | 95.74 | 99.32 |
|
| 33 |
+
| Entity Matching | DBLP-GoogleScholar | 95.60| 76.12 | 91.87 | 92.83 | 89.45 | 95.16 |
|
| 34 |
+
| Entity Matching | Amazon-Google | 75.58| 66.53 | 74.21 | 72.69 | 56.64 | 80.25 |
|
| 35 |
+
| Data Imputation | Restaurant | 77.20| 94.19 | 97.67 | 94.19 | 93.02 | 93.02 |
|
| 36 |
+
| Data Imputation | Buy | 96.50| 98.46 | 100 | 100 | 100 | 100 |
|
| 37 |
+
| Error Detection | Hosptial | 99.10| 90.74 | 90.74 | 92.21 | 65.66 | 86.59 |
|
| 38 |
+
| Error Detection | Adult | 94.40| 92.01 | 92.01 | 96.62 | 90.13 | 99.20 |
|
| 39 |
+
| Schema Matching | Sythea | 38.50| 57.14 | 66.67 | 36.36 | 30.77 | NA |
|
| 40 |
|
| 41 |
_Accuracy as the metric for data imputation and the F1 score for other tasks._
|
| 42 |
_For GPT-3.5, GPT-4 we used the few-shot approach, while for Jellyfish and Jellyfish-Reasoning, the zero-shot approach was employed._
|
|
|
|
| 47 |
[HoloClean](https://arxiv.org/abs/1702.00820) for Data Imputation
|
| 48 |
2.
|
| 49 |
[Large Language Models as Data Preprocessors](https://arxiv.org/abs/2308.16361)
|
| 50 |
+
3. Jellyfish-13B-1.1 is set to be the next iteration of Jellyfish-13B and is presently under development.We're showcasing its performance at this stage to highlight its impressive potential.As demonstrated in the table, it has already outperformed Non-LLM methods on the majority of benchmark datasets. We've optimized the training data for this 1.1 version, and its release is on the horizon.
|
| 51 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
|
| 53 |
**Jellyfish paper will be coming soon!**
|
| 54 |
|