Why AI gives confident wrong answers

An assistant of mine confidently gave a wrong number to an insurance advisor. The model was fine. The library it read from was not, and that is where most AI projects actually die.

Last week an assistant I built for an insurance unit was asked how many conditions one of their critical illness policies covers. It answered nineteen. It then listed all nineteen, alphabetically, ending at Multiple Sclerosis.

The real answer is twenty three.

No model was hallucinating. The system had been given the company's own material, and somewhere in that material was a scanned slide whose list had been cut off at the bottom of the page. It stopped at M. Another document in the same library stated the correct total of twenty three and listed every one, including the four the scan had lost.

So the library contained both the right answer and a truncated one, and which answer an advisor got depended on which page the system happened to pull. An advisor could have quoted nineteen to a client. Nobody would have caught it, because the answer looked complete. It had a list. Lists look like evidence.

I found it because I asked the question four times and compared, not because anything failed.

This is the actual failure mode

The industry keeps describing AI failure as the model making things up. In business systems that is rarely what goes wrong. What goes wrong is that the material the model was given is incomplete, contradictory, out of date, or three versions of the truth that nobody reconciled, and the model does exactly what it was asked to do with it.

This is not a niche opinion any more. Reporting through 2026 puts data quality, rather than model quality, as the single biggest cause of automation failure, with something like 60 percent of AI projects failing on data quality alone, and Gartner expecting more than 40 percent of agent projects to be cancelled by 2027 once the effort involved becomes obvious.

Put plainly: automation does not fix a mess. It industrialises it. A person reading that truncated slide would have thought "this page is cut off". Software has no such instinct unless you build one.

What I changed, in order

I fixed the document, not the prompt. The temptation is to write a cleverer instruction. I restored the missing four items from the authoritative source, marked in the file where they came from, and rebuilt the index. Four runs, four correct answers.

I made truncation survivable. The system now retrieves the passage either side of every match, because a list that spans a page break was arriving with its head cut off. It was reading the tail of a list and counting what it could see.

I gave it a rule about counting. If the material states a total, use the stated total, even when the visible list is shorter. Never count the items in front of you and present that as the answer. That single rule turns a confident wrong number into a correct one.

I stopped it from blending questions. Ask about a hiring form, then about a product, and the retrieval was mixing both questions into one search, so the product answer came back holding hiring pages and half the product page. Each question is now searched on its own unless it genuinely cannot stand alone.

None of that is model work. It is librarian work.

The tell that your material is the problem

Ask your system the same question three or four times in a fresh session each time. Not variations. The same question.

If the answers differ in substance, you do not have a model problem. You have two sources that disagree, and the system is picking between them at random. Go and find them. In my case it took one search across the library to find the second, shorter list.

While you are there, ask it something you know is not in the material at all. A system that answers anyway is a system that will answer anyway about your prices.

Why this matters more in a small business

A large company has a team whose job is to notice that a report looks odd. A clinic has a receptionist with four people at the desk. When the screen says nineteen, that is what gets said to the client, and the error travels at the speed of confidence.

The upside is that a small business can actually fix this. The material is a few dozen documents, not a data warehouse. An afternoon spent working out which document is authoritative, deleting the superseded ones, and repairing the scans that lost their bottom line will do more for the accuracy of an AI system than any amount of switching models.

What I now do before switching anything on

• One authoritative source per fact. If two documents state a price, one of them gets deleted, not kept "for reference".
• Every scanned page checked to see that it ends where the original ends. Cut off pages are the commonest defect, and the hardest to notice.
• The same question asked repeatedly, in fresh sessions, before anyone else is allowed near it.
• A written rule that the system must hand off rather than answer when the material does not cover the question.
• Someone reads the transcripts for the first month. Every business I have done this with found two or three bad answers, and every one was fixed by repairing a document.

The nineteen was not a clever failure. It was a photocopy that ran out of page. Most of the AI problems you will actually meet look like that, and they are fixable in an afternoon by somebody willing to read the source material.

Sources: SR Analytics on data quality and AI failure, Kaizen AI Consulting on 2026 agent projects.