The Chat Always Answers, and That's the Problem

A chat on top of your warehouse feels like you’ve finally crossed the finish line. Users ask in plain language, the model generates SQL, and the endless queue to your data team simply disappears.

Then someone asks how many active customers you had last quarter. They get a number back. They carry it into a meeting. Nobody can say what it counted as a customer, or as active, or whether it left out test accounts.

The chatbot does not fail at language. It fails at meaning.

The failure doesn’t look like an error, it looks like a number

A system that doesn’t understand a question has two moves: refuse, or invent. A language model almost always invents, and it does so with flawless prose.

dbt Labs frames it this way: with text-to-SQL, failure looks like a plausible but wrong answer. A declared layer fails by returning an error message instead.

That distinction is real. Worth knowing it comes from dbt Labs, though—they sell the declared layer, and the page that makes this distinction is their own benchmark running itself against text-to-SQL. I’ll return to this.

Still, the distinction holds even if you trust zero numbers from anyone. An error stops someone and gets fixed. A plausible number slides into a deck and spreads.

Asking is not agreeing

Natural language questions do not make modeling decisions disappear. They push those decisions into a place where nobody sees them happen.

flowchart TD
    Q["How many active customers<br/>last quarter?"] --> M{"Is 'active customer'<br/>declared anywhere?"}
    M -- yes --> S["Resolved against entity,<br/>grain and filters"]
    M -- no --> G["Inferred from column names<br/>and a best guess"]
    S --> A["One answer, or an error"]
    G --> P["A plausible number"]

The left branch only exists if someone already wrote what an active customer is. If nobody did, the model does not pause to ask: it makes the choice itself, and that choice is not recorded anywhere.

This is the same problem I described in the ontology of the semantic layer, but with a difficult difference. In that post, disagreement eventually surfaced because two teams reported different revenue numbers.

Here each person asks alone, in their own chat window, and no one compares.

What the benchmarks measure, and what they don’t

Worth examining numbers that were measured with care, and knowing what they do and don’t show.

The BIRD benchmark, presented at NeurIPS 2023 and built across large, messy databases, showed ChatGPT achieved 40.08% execution accuracy compared to 92.96% for annotators. Before that, Spider identified the real hard part: not writing SQL, but generalizing to schemas the model has never encountered.

Today’s models score better than those. But here is the subtle point that matters: these benchmarks do not measure what this post is about. They compare the generated query against gold-standard SQL written by an annotator, which means someone already resolved what the question meant.

The ambiguity in “active customer”—that is exactly what the measurement leaves out. The lab tests whether the model translates a settled intention correctly. Inside your company, nobody settled the intention yet.

An agent cannot reason about what nobody wrote

The answer is not to ban the chat. It is to write down the definition somewhere, and make the agent read that file.

# The question from the beginning, resolved against something written.
metric:
  name: active_customers
  entity: customer
  grain: "one row per account, not per contract"
  filters:
    - "status = 'active'"
    - "is_test = false"
  time_dimension: subscription_period

Ten lines that no chat can reinterpret three different ways. With this in place, the question from the first paragraph has exactly one answer.

flowchart LR
    Q["active customers<br/>last quarter"] --> E["entity: customer"]
    E --> G["grain: one per account"]
    G --> F["filters: active,<br/>not test"]
    F --> T["time: subscription_period"]
    T --> A["One resolved query"]

Every step in that chain is a choice somebody made once. Without the file, the model makes all five choices itself, and can make them differently each time.

Worth saying where this half comes from. Cube and Malloy document this approach, and both sell products that implement it. Open Semantic Interchange is a vendor-neutral standard, though “neutral” here means neutral among vendors: it is backed by Snowflake, dbt Labs, Databricks, and Salesforce.

Four of this post’s seven sources sell the solution it describes. I do not know of any peer-reviewed independent measurement of how much a declared layer improves accuracy— including the dbt benchmark I cited. Judge the argument by how it works, not by the numbers behind it.

Lloyd Tabb, who created LookML and Looker before Malloy, makes his case in a research talk, not a product pitch.

He argues that SQL forces you to rebuild meaning from scratch every time, and writing it once changes who has to understand it.

Failure modes worth naming

The pattern across all six: the model was told to choose something the team never agreed on.

Where to start

Do not enable chat yet. Find your three most-asked questions in Slack and write out each one: which entity answers it, what is the grain, what are the filters.

If writing them down shows disagreement between team members, you have just discovered what the chat would have answered too—without warning anyone.

References

1. dbt Labs — Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update, dbt Developer Blog
2. Cube Dev — Introduction, Cube Documentation
3. Jinyang Li et al. — Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs, NeurIPS 2023
4. Tao Yu et al. — Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task, EMNLP 2018
5. Malloy Data — Querying a Semantic Model, Malloy Documentation
6. Open Semantic Interchange — Introduction, OSI specification
7. Lloyd Tabb — Malloy: A Modern Open Source Language for Analyzing, Transforming, and Modeling Data, CMU Database Group, 2025