---
license: apache-2.0
datasets:
- PipableAI/pip-txt-to-sql-spider-bird-dataset
language:
- en
metrics:
- accuracy
tags:
- sql
- code
- text2sql
- instruction_tuned
- basemodel
- jax
- pytorch
- tensorflow
- text-generation-inference
library_name: transformers
pipeline_tag: text-generation
---
# pipSQL-1.3b
[pipableAi](https://www.linkedin.com/company/pipable.ai/about/)

## What have we built?
A 1.3 bn SQL model that outperforms most SQL expert models and chatgpt on popular benchmarks.
This is a distilled model built on the deepseek base model.

## How we built it?

We used softmax cross entropy and a modified form of policy grad along with Q loss, optimized in an EM set up.

## Benchmarking :
For benchmarking purposes we are using Semantic Evaluation for Text-to-SQL with 
Distilled Test Suites, an officially accepted evaluation framework for Spider, SParC, and CoSQL which was proposed by a research team of Yale and Berkeley. 
The benchmark contains 2200 test data points
Here is the link to run the evaluation:


[Test Suite SQL Eval](https://github.com/taoyds/test-suite-sql-eval)

|model|easy|medium|hard|extra|
|-----|----|------|----|-----|
|sqlcoder-7b-2|72.0|58.0|40.6|37.3|
|pip-sql-1b-Qstar|74.0|54.0|36.5|30.0|
|pipSQL-7b|63.0|40.0|30.2|25.0|
|sqlcoder-7b|60.6|48.2|28.3|20.4|
|gpt-3.5|58.8|44.7|31.0|28.4|

We have also benchmarked it on defog eval.
It contains 200 test data points handpicked by defog team.
Here is the link to it:


[Defog SQL-Eval](https://github.com/defog-ai/sql-eval)
These are the results -

![image/png](https://cdn-uploads.huggingface.co/production/uploads/64d32c6b921678fdc9de3302/ZU_YeKLEtHg7ImXI5LrXo.png)

## License
The model is open source under apache 2.0. License

## Usage

### Installation

```bash
pip install transformers
```

### Prompt
```python
prompt = f"""<schema>{schema}</schema>
<question>{question}</question>
<sql>"""
```

### PyTorch
```python
from transformers import AutoModelForCasualLM, AutoTokenizer
device = "cuda"
model = AutoModelForCausalLM.from_pretrained("PipableAI/pipSQL-1.3b")
tokenizer = AutoTokenizer.from_pretrained("PipableAI/pipSQL-1.3b")

inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True).split('<sql>')[1].split('</sql>')[0])
```

### Flax
```python
from transfomers import FlaxAutoModelForCausalLM, AutoTokenizer
device = "cuda"
model = FlaxAutoModelForCausalLM.from_pretrained("PipableAI/pipSQL-1.3b")
tokenizer = AutoTokenizer.from_pretrained("PipableAI/pipSQL-1.3b")

inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True).split('<sql>')[1].split('</sql>')[0])
```

### TensorFlow
```python
from transfomers import TFAutoModelForCausalLM, AutoTokenizer
device = "cuda"
model = TFAutoModelForCausalLM.from_pretrained("PipableAI/pipSQL-1.3b")
tokenizer = AutoTokenizer.from_pretrained("PipableAI/pipSQL-1.3b")

inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True).split('<sql>')[1].split('</sql>')[0])
```