Why conversational AI with your data often fails and how proper data governance can transform the outcome. Learn how context is key to unlock AI.
For the last couple of years, the industry has been obsessed with a simple idea: you ask a question in plain English, an agent understands your intent, generates SQL, queries your systems, and gives you exactly what you need.
The marketing demos are always the same. Someone types: “What’s the probability of default for our Retail customers in Q3?”
The AI produces a query in a few seconds. It looks clean, elegant, almost reassuring. You start to believe the future has finally arrived.
Then you look closer.
The dataset the model picked is not the one your Risk team considers authoritative. The formula for “probability of default” is not the one approved internally. The segmentation behind “Retail” is outdated. The fields look right but mean something else entirely. And the table the AI chose happens to be a near-duplicate of another dataset that no one has touched in years but that still sits in production because no one has the courage (or time) to deprecate anything.
Suddenly, the magic evaporates.
And at this point, most people instinctively blame the AI: “It didn’t understand the question.” “It hallucinated.”
But the model didn’t hallucinate at all. It did exactly what your data estate told it to do.
It produced a query based on the information you gave it — and that information lacked meaning.
This is the part no one likes to say out loud: LLMs don’t fail at SQL. They fail at everything around the SQL — the business meaning, the formulas, the definitions, the boundaries, the semantics, the context. And none of that lives inside the database. It lives in people’s heads, in documents no one updates, in PowerPoints long forgotten, in pieces of tribal knowledge circulating informally between teams.
The reason AI struggles is brutally simple: the organization has never written down the logic it expects the AI to understand. We want the model to “behave like an analyst”, but we give it the inputs of a photocopier.
In theory, all the missing context should be produced by people. In practice, this almost never happens. And it’s not because people don’t care — it’s because the reality inside enterprises pushes everyone to ship something that works “well enough”, explain it later, move to the next initiative, and survive the quarter.
The conversational use case is interesting precisely because it exposes this dynamic in the most brutal way possible.
If you really want to see the maturity of a data organization, don’t run a survey.
Ask a model a real business question and see what happens.
Within sixty seconds, all the weaknesses emerge: duplication, semantic drift, inconsistent definitions, obsolete datasets, missing lineage, outdated formulas, absence of examples, and that peculiar form of ambiguity that everyone feels but no one can quite articulate.
So the real question is: if people cannot reliably produce the context required for the business to operate, how can we expect AI to do it?
The answer is that we shouldn’t — at least not directly.
The context must still come from people, but the way it is produced needs to change. It cannot rely on goodwill, memory, or discipline. It cannot be optional, manual, or left to interpretation. In other words, it cannot be the side-effect of the process. It must become part of the process itself.
This is where governance needs to shift from “reviewing after the fact” to “shaping the process before the fact.” People don’t need policing; they need an environment that nudges them in the right direction and quietly stops them when they’re about to create future problems.
A system that says: “This description isn’t meaningful, rewrite it.” A system that notices that the dataset you’re about to publish overlaps almost entirely with another one.
A system that knows which formula is supposed to be used and which definition of “Retail” is actually valid. And most importantly, a system that doesn’t wait for someone to approve anything manually, but integrates all of this directly into the lifecycle.
This is the essence of computational governance. It’s not about control — it’s about guidance. It’s about creating an environment where producing context becomes natural, even effortless, because the platform itself points out what’s missing and what’s inconsistent, long before anything reaches production.
And when something is too far off, it simply doesn’t go through. Not because someone said no, but because the system itself cannot accept something incoherent with the rules of the ecosystem.
Once this context starts to exist — definitions, formulas, examples, semantics, ownership, intent — something interesting happens. You ask the AI the same question again, and the output changes completely.
The model no longer has to guess what “probability of default” means. It no longer has to infer which dataset is official, or how segmentation works, or which field represents the real signal. It simply uses the meaning the organization has finally formalized.
And at that moment, “chat with your data” stops being a slide in a vendor keynote and becomes an actual capability.
This is the part people underestimate: AI is not the blocker. Missing context is and context does not write itself and will not be produced by a tool. It must be produced by people, but it must also be enforced — gently, consistently, and automatically. Otherwise, it will always be the first thing sacrificed under pressure.
If you want conversational access to data, you don’t start from the AI.
You start from the discipline that ensures the AI has something meaningful to work with.
Everything else is just a very expensive Demo or POC.
Data Mesh is completely changing the perspective on how we look at data inside a company. Read about what Data Mesh and how it works.
Data Mesh is completely changing the perspective on how we look at data inside a company. Read what is Data Mesh and how it works.
Discover the potential of data contracts in Witboost and how they enhance data quality, governance, and interoperability.