Why your AI tool gives vague answers (and why a better prompt will not fix it)

You have put Copilot or ChatGPT to work. Someone asks whether a particular customer can have sixty days to pay. Out comes a neatly worded answer. It just is not your answer. And however you rephrase the question, it does not get any better.

This is not a teething problem, and it will not go away with the next model release. It is structural, and it sits somewhere other than where most companies go looking for it.

There are two kinds of question in your company

The first is a search question: where is the contract with Vermeulen? A document exists for that. An AI tool finds it, opens it, and quotes from it. This works well, and it is what Copilot, Glean and most AI assistants were built for.

The second is a decision question: can Vermeulen pay in sixty days? Can I give this discount? Is this damage still under warranty? No document exists here. The agreement was made at some point, in a conversation or an email thread, and never recorded anywhere as a rule that applies.

Your AI tool knows everything about the world and nothing about your company. Not because it is stupid, but because nobody told it what applies at yours.

What happens when you ask anyway

The tool does what it always does: it looks for something that resembles an answer. And it finds plenty. An email thread from fourteen months ago in which an account manager agreed to sixty days. General terms from 2023 that say thirty. An undated internal procedure with no author that says prepayment only. A 2024 quotation with an exception of forty-five days.

Four sources, all four true at some point. None of them carries any authority, and nothing says which one wins.

The model picks one, and does not tell you it made a choice. No warning, no caveat, no note saying it found contradictory information. You get a fluent, confident answer that may well be wrong.

Why a better prompt does not help

This is where most companies get stuck. Someone organises a prompting course, someone takes on prompt engineering, templates appear. And it does not get better.

Prompting decides how a model answers: the tone, the length, the format. It does not decide what the model has to work with. If the agreement was never recorded, a better prompt changes the shape of the answer, not whether it is right. You get the same wrong answer, better written.

Why this is getting worse, not better

In the old situation, one employee gave one wrong answer to one customer. That was usually caught somewhere along the way: a colleague thought it through, someone called back, the account manager said it was different.

Now one tool gives that same answer to everyone who asks. For months. With complete confidence and no caveat. And nobody is thinking along any more, because people barely check what a system tells them. That is not carelessness. It is simply how people deal with systems.

The error only surfaces once it has become expensive: on an invoice, at a customer, or during an audit.

Where the answer has to come from instead

Not from a better model. From the people who know, and from the documents where it is half recorded.

In practice that means gathering the agreement, having it confirmed by the person who knows it and by the person accountable for it, and then recording it with three things that almost never appear anywhere:

• An owner. A name, not a department. Someone responsible for this being right.
• A date. When was this approved, and when will it be reviewed?
• A source. Where does it come from? Which agreement, which decision, which arrangement?

Only once that is in place can your AI tool do anything sensible with it. And you do not need to replace anything: the same tool, with your agreements in it, gives an answer that is right, and shows who approved it and when.

What you can do tomorrow

Ask one question to three different people in your own organisation. Choose something with money attached: the maximum discount, the standard payment term, the warranty period after handover.

If you get the same answer three times with the same reasoning, you do not have this problem. If you get three different answers, or the same answer three times without anyone being able to say where it is written, you know enough. And you also know why your AI tool cannot perform better than the people feeding it.