Why a chatbot on your own documents does not work the way you hoped
I build these systems. Ingest the documents, chunk them, create embeddings, put a vector database behind it and a language model in front. Technically it is now an evening's work. And yet almost every client comes back with the same question: why is the answer wrong?
The honest answer is uncomfortable for my own field. It is not the chunking, not the embedding model and not the reranker. It is something no architectural choice can get around.
What retrieval actually does
When someone asks what the maximum discount is for a particular customer, this happens: the question is turned into a vector, and the system looks for the passages whose vectors sit closest to it. That is a similarity measure, not a judgement.
What comes back is four fragments. A first-quarter price list. A sales day presentation. A message from the sales lead. A sales handbook from 2021. All four are about discounts, all four closely resemble the question, and that is exactly why all four score high.
Semantic similarity measures what something is about. It does not measure whether it is true, whether it still applies, or who approved it.
The model gets those four fragments in its context and has to turn them into one answer. There is no field from which it can read which one applies, because that field does not exist. So it does what it always does: it produces the most likely answer, fluently phrased, without a caveat.
The four fixes that do not work
A larger context window
The idea: just put everything in, then the model sees the full picture. In practice it means the model sees more contradictory sources at the same time. It still does not know which one wins, and the chance of a wrong choice goes up rather than down.
A better model
A stronger model reasons better about the information it has. If the information needed to choose is missing, the answer does not become more correct, only more convincingly phrased. That is a step backwards here: a hesitant answer could at least have warned you.
A knowledge graph
I have built them myself, and they are beautiful. They organise what exists: which document belongs to which customer, which product, which process. But a graph of outdated documents is a neatly organised collection of outdated documents. A CTO I know put it this way: the graph works perfectly, the information is still wrong.
Better prompts
"Answer only on the basis of the most recent source." Sounds sensible. But the model does not know which source is most recent unless that appears as data in the text. And most company documents contain no reliable date, let alone an approval date.
What is missing is metadata, not intelligence
To decide which of four answers applies, you need exactly three things. They are boring, they are not technically complicated, and they are almost never recorded anywhere:
- Who approved this. A name, not a department. A department cannot approve anything.
- When. And with it: when will it be reviewed? Without a review moment, everything quietly decays.
- Where it comes from. Which agreement, which decision, which arrangement, so the answer can be checked.
Once those three fields sit with every rule, the problem suddenly becomes trivial. Two sources contradict each other? Then the one with the most recent approval by the accountable owner wins. That is no longer an AI problem but a sorting problem.
Why this cannot be automated
The obvious follow-up question is whether you could simply have a model generate that metadata. You can, and it is a bad idea.
A model can guess who probably approved something. That guess looks exactly like a fact: the same formatting, the same certainty. From then on it can no longer be told apart from a fact. You have not added governance; you have created a second layer of unreliability and made it look official.
That is why we build it the other way round. The model may structure, summarise and make suggestions. It may never quietly resolve a contradiction. When the system finds two versions that contradict each other, it flags both, quotes both, and hands the choice to a person. That choice is recorded, including who made it.
It is slower. It is also the only way you can later explain why it said what it said.
What the technology is good for
I am not against retrieval. I use it every day. It is excellent for finding things: where is this contract, which version of this document is the latest, where did we discuss this before.
But there is a category of questions that has no document attached. "Can this customer pay in sixty days." That agreement was made, not written. And you cannot index what was never written down, however good your architecture is.
For that category you first have to do something that is not technical: gather it from the people who know, have it confirmed by whoever is accountable, and record it with an owner, a date and a source. Only then does your retrieval have something to work on that can carry an answer.
How to test this yourself
If you have a chatbot running on your own documents, ask it a decision question with money attached. Not "where is X" but "may X". Then look at the sources it cites and ask yourself three things:
- Do I know for every source when it was approved?
- Do I know who is responsible for it?
- When two sources contradict each other, did the system tell me?
Answer the three questions.
If the answer is no three times, your problem is not the architecture. It is what sits underneath.
Want to know where you stand?
Half an hour with two of us. Afterwards you get a single page: the agreements most at risk at your company, and who should own them. You keep that page even if we go no further.
Book the check →