Skip to content

AI that confidently talks nonsense: check, compare and admit doubt

Language models sometimes make up a source or an amount, and state it just as firmly as a correct answer. You cannot prevent that, but you can control it: with checks, comparison and a system that dares to admit doubt.

The problem

A language model predicts words. Usually that produces a good answer, but sometimes it makes up a fact, a source or an amount, with the same conviction. This is called hallucinating. It happens everywhere: courts around the world have already dealt hundreds of times with filings that contained invented rulings or sources.

Why it keeps coming back

  • A model does not know what it does not know.
  • Too much or too little context increases the chance of errors.
  • An answer sounds just as certain whether it is right or not, so the error goes unnoticed.

Our approach

  • Provide the right information. Not the whole file, but the few relevant facts, with their source. That reduces the chance of fabrication.
  • Calculation and lookup by code. Amounts, dates and facts come from the source, not from the model's memory.
  • Have answers checked. A model can give different answers to the same question. We have another model check those answers. That gives us a cross-section. For numerical answers, their average can give a picture.
  • Escalate when needed. If the check is not certain, the system looks for more evidence or has a heavier model review the answer.
  • Admit doubt honestly. If doubt remains, the system says so instead of guessing.

What to watch out for

  • Always ask for sources, and check them for decisions that matter.
  • Let a person decide where errors cost money or reputation.
  • Measure quality with a fixed set of test questions, so you can see whether a change makes things better or worse.

Would you like advice on reliable AI? Request a free advice conversation.

Sources

Updated on

AI

Personal data in the AI cloud: who is watching?

A conversation with an AI service in the cloud is not private by default. A case in Canada shows what a provider can see and share. What does that mean under the GDPR, in the United Kingdom and in Switzerland?