license: apache-2.0
datasets:
- PipableAI/pip-txt-to-sql-spider-bird-dataset
language:
- en
metrics:
- accuracy
tags:
- sql
- code
- text2sql
- instruction_tuned
- basemodel
- jax
- pytorch
- tensorflow
- text-generation-inference
library_name: transformers
pipeline_tag: text-generation
pipSQL-1.3b
What have we built?
A 1.3 bn SQL model that outperforms most SQL expert models and chatgpt on popular benchmarks. This is a distilled model built on the deepseek base model.
How we built it?
We used softmax cross entropy and a modified form of policy grad along with Q loss, optimized in an EM set up.
Benchmarking :
For benchmarking purposes we are using Semantic Evaluation for Text-to-SQL with Distilled Test Suites, an officially accepted evaluation framework for Spider, SParC, and CoSQL which was proposed by a research team of Yale and Berkeley. The benchmark contains 2200 test data points Here is the link to run the evaluation:
| model | easy | medium | hard | extra |
|---|---|---|---|---|
| sqlcoder-7b-2 | 72.0 | 58.0 | 40.6 | 37.3 |
| pipSQL-1.3b | 71.1 | 49.9 | 31.5 | 24.1 |
| pipSQL-7b | 63.0 | 40.0 | 30.2 | 25.0 |
| sqlcoder-7b | 60.6 | 48.2 | 28.3 | 20.4 |
| gpt-3.5 | 58.8 | 44.7 | 31.0 | 28.4 |
We have also benchmarked it on defog eval. It contains 200 test data points handpicked by defog team. Here is the link to it:
Defog SQL-Eval These are the results -
License
The model is open source under apache 2.0. License
Usage
Installation
pip install transformers
Prompt
prompt = f"""<schema>{schema}</schema>
<question>{question}</question>
<sql>"""
PyTorch
from transformers import AutoModelForCasualLM, AutoTokenizer
device = "cuda"
model = AutoModelForCausalLM.from_pretrained("PipableAI/pipSQL-1.3b")
tokenizer = AutoTokenizer.from_pretrained("PipableAI/pipSQL-1.3b")
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True).split('<sql>')[1].split('</sql>')[0])
Flax
from transfomers import FlaxAutoModelForCausalLM, AutoTokenizer
device = "cuda"
model = FlaxAutoModelForCausalLM.from_pretrained("PipableAI/pipSQL-1.3b")
tokenizer = AutoTokenizer.from_pretrained("PipableAI/pipSQL-1.3b")
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True).split('<sql>')[1].split('</sql>')[0])
TensorFlow
from transfomers import TFAutoModelForCausalLM, AutoTokenizer
device = "cuda"
model = TFAutoModelForCausalLM.from_pretrained("PipableAI/pipSQL-1.3b")
tokenizer = AutoTokenizer.from_pretrained("PipableAI/pipSQL-1.3b")
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True).split('<sql>')[1].split('</sql>')[0])