richardr1126 commited on
Commit
16179f5
·
1 Parent(s): b041bb7

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -0
README.md CHANGED
@@ -26,6 +26,18 @@ The goal of the Spider challenge is to develop natural language interfaces to cr
26
 
27
  This dataset was used to finetune this model.
28
 
 
 
 
 
 
 
 
 
 
 
 
 
29
  ## Citations
30
 
31
  ```
 
26
 
27
  This dataset was used to finetune this model.
28
 
29
+ ### Spider Skeleton WizardCoder - [test-suite-sql-eval](https://github.com/taoyds/test-suite-sql-eval) Results
30
+ With temperature set to 0.0, top_p set to 0.9, and top_k set to 0, the model achieves **61% execution accuracy** on the Spider dev set.
31
+
32
+ <img src="https://raw.githubusercontent.com/cuplv/text-to-sql-wizardcoder/main/eval/plots/spiderwizard-plus-chatgpt.svg" height="300">
33
+ <img src="https://raw.githubusercontent.com/cuplv/text-to-sql-wizardcoder/main/eval/plots/spiderwizard-vs-chatgpt.svg" height="300">
34
+
35
+ Note:
36
+ - ChatGPT was evaluated with the default hyperparameters and with the system message `You are a sophisticated AI assistant capable of converting text into SQL queries. You can only output SQL, don't add any other text.`
37
+ - Both models were evaluated with `--plug_value` in `evaluation.py` using the Spider dev set with database context.
38
+ - `--plug_value`: If set, the gold value will be plugged into the predicted query. This is suitable if your model does not predict values. This is set to `False` by default.
39
+
40
+
41
  ## Citations
42
 
43
  ```