Why "Talk to Your Data" Usually Doesn't Deliver

The demo lands. Someone speaks into a phone: "What was our revenue in the second quarter?" Two seconds later, a number. The room nods.

The problem sits below the surface. In most of these systems you aren't talking to your data. You're talking to a story about it. Those are not the same thing.

RAG is brilliant for knowledge. For data it's the wrong tool.

Almost every "talk to your data" product runs on RAG - retrieval-augmented generation. Your documents get chopped up, turned into vectors, dropped into a database. You ask. The system finds the most similar passages. A language model turns them into an answer.

For knowledge, that works. A policy, a process, a handbook: perfect.

For data, it doesn't. The reason is mechanical. Better prompting changes nothing.

RAG searches for similarity. Not truth. It finds the passage that fits best - not the correct record. If your Q2 revenue sits in an April PDF that has since been corrected twice, RAG pulls the old number. It doesn't know the number is stale. It only knows it looks similar.

The test you can run today

Ask the same system the same question about quarterly revenue three times.

You may get three answers. The data didn't change. Retrieval is probabilistic: these fragments, then those, phrased anew each time.

A database query filtered by entity, period and currency returns the same result every time. RAG returns probabilities that feel right.

For "explain our travel expense policy," that's fine. For "how much does customer X still owe us," it isn't.

It gets worse. RAG guarantees no filters. Periods, entities, currencies, permissions: hard conditions in the real world. RAG treats them as soft hints in the text. It will hand you the revenue of the wrong subsidiary, in the wrong quarter, in the wrong currency - without blinking.

And the wrong answer sounds every bit as confident as the right one. Nobody can hear whether a spoken number came from a live query or from a fragment two quarters old.

The reflex - bigger index, better prompting, more examples - doesn't help. The problem isn't volume. It's the nature of the query. A similarity search never becomes a binding one. You can question a witness who remembers something similar for as long as you like. Their memory won't get more precise. Only longer.

The price scales with the stakes. On an internal note, an approximate number is harmless. On an investment approval, it's a liability.

The alternative that holds: RAG for knowledge, tool calls for data

The answer isn't "more documents." It's separation - by the nature of the question.

Knowledge lives on explanation and context. Policies, processes, product know-how, FAQs. RAG was built for that.

Data lives on currency and precision. Revenue, inventory, open items, pipeline. That needs a tool call: a real, filtered, authorized query against ERP, CRM or BI. Not pre-processed text. The data itself.

The voice intelligence decides per question: explain or query. Between them sits a policy layer. It enforces permissions, entities, masking and audit.

The result has three properties RAG alone can't deliver. Reliable. Traceable. Reproducible. The same question yields the same answer. And afterwards you can show where it came from.

Where we stand

We're not the mouthpiece of a tool vendor. We won't sell you the system that shines in the demo and shifts the liability in production.

The architecture that separates knowledge from data comes out of our partnership with Overmind. Our job is the less comfortable one: deciding with you whether, when and what for voice makes sense in your stack. And where you're better off leaving it alone.

Voice without that architecture is a party trick. Voice with it carries operational decisions. The difference costs you money at exactly the moment the answer stops impressing people and starts flowing into a number, a contract or a dispatch.

Your next step

Before you invest in a "talk to your data" project, get your bearings. How robust would voice AI be on your stack? Where does knowledge sit, where does data? And which of it is clean enough to be queried and owned by a spoken answer?

That's exactly what our Voice AI Readiness Check examines. In a few minutes you see where you stand - and which gaps to close before the first spoken answer feeds a real decision.