AI-written SQL
We let an AI write SQL from plain questions and checked every answer against queries we wrote ourselves.
The question
Most company data sits in relational databases, and getting it out takes SQL. Tools that turn plain language into queries promise to remove that barrier. We wanted to know how mature they really are, and what decides whether they get it right.
What we built
An SQL agent built with LangChain. GPT-4o receives the question, the relevant tables as CREATE statements with three sample rows, and five example queries picked by semantic similarity. The agent checks its query before running it and corrects itself after an error. It only has read access, and its temperature is set to zero, so the same question always gets the same answer.
How we tested it
We wrote a set of SQL queries by hand for the Chinook database, a well-known sample of a digital music store, and ranked them by difficulty. For each one we phrased a question in plain language. An answer counted as correct if its output matched ours, however the SQL got there.
What came out
84%
Chinook, likely known to the model.
77%
Car dealer, a database we built ourselves.
Most answers were right on the first try. The misses were among the hardest queries, so we expected the SQL to be the problem. It was not. Each time, a more precise question fixed it.
One example: we asked the agent to group customers by spending and to “sort the customers by these categories”. It sorted them by customer number instead. Changing “the customers” to “the list” gave the right result.
The SQL itself followed the standards but was inconsistent. The agent switched between naming styles and used table aliases in some joins but not in others.
On the database the model had never seen, accuracy was a little lower. Whether its familiarity with Chinook explains the gap would take a larger test to say.
Our conclusion: the tool lowers the barrier, but someone still needs to know SQL to judge whether an answer is right.
Decisions
Build on the official guide. We followed LangChain’s own tutorial instead of writing everything from scratch, so anyone can rebuild the setup and our results are not shaped by our own detours.
Borrow the example queries. The examples the model learns from came from the guide, not from us. Examples we wrote ourselves would have resembled our test questions and flattered the result.
Test on a database the model has never seen. Chinook appears in countless tutorials, so the model may know it. We built a small car dealer database and adapted our queries to it, to see whether the model understands data or remembers it.