Skip to content

How to Prevent AI Hallucinations in Your Business (and Keep Your AI Honest)

By USA AI Corp · · 6 min read

The short answer

You can't make a language model incapable of inventing things, but you can build the system around it so made-up answers are rare and easy to catch. Make it answer only from your own documents, quote the passage it used, decline when the sources don't cover a question, send low-confidence answers to a person, test it on real questions before launch and review its misses every week.

On March 1, 2023, a New York law firm filed a brief in a personal injury case against the airline Avianca. Among its citations were six court decisions, including one called Varghese v. China Southern Airlines, that did not exist. One of the lawyers had found them using ChatGPT.

The detail worth remembering comes from the judge's later opinion. The lawyer had asked ChatGPT directly, 'Is Varghese a real case,' and 'Are the other cases you provided fake.' ChatGPT replied that its authorities were real and could be found on Westlaw and LexisNexis.

In June 2023, Judge P. Kevin Castel of the Southern District of New York sanctioned the two lawyers and their firm, imposing a $5,000 penalty and requiring letters to each judge falsely named as the author of a fake opinion. He also wrote that there is 'nothing inherently improper about using a reliable artificial intelligence tool for assistance,' but that lawyers have a gatekeeping role to make sure their filings are accurate.

The lesson for any business isn't to avoid AI. It's that asking the model whether it's sure is not a check. The check has to come from outside the model: a source a person can open and read. (The case is told here as a cautionary example, not as legal advice, and professional rules on AI use differ from state to state.)

Why a business AI makes things up

A language model writes by predicting likely next words. Most of the time that produces accurate, fluent text. When the model lacks the facts, it still produces fluent text, and fluent text sounds sure of itself.

NIST, the federal standards agency, calls this 'confabulation' in its 2024 profile of generative AI risks. It notes that models can also produce made-up citations and reasoning that make a wrong answer more convincing. NIST's own example comes from health care: a confabulated summary of patient information could lead a doctor to the wrong diagnosis. In a law firm, the equivalent is a deadline or a citation.

The risk is highest where a wrong answer costs the most. That's why the fixes below are about the system around the model, not about hunting for a more truthful model.

Six controls that keep answers honest

1. Answer only from your sources

Connect the assistant to the documents that hold your answers (protocols, policies, templates and, where access rules allow, the matter or chart) and instruct it to answer only from what it retrieves. Answering only from your sources does the most. It reduces errors without eliminating them: in a preregistered Stanford evaluation, legal research tools from LexisNexis and Thomson Reuters built this way still hallucinated on between 17% and 33% of test queries, fewer than a general chatbot but far from zero. Patient charts belong only with a vendor under a Business Associate Agreement; our guide to using AI with client data covers those conditions.

2. Quote and link every answer

Each answer should show the passage it relied on, with a link to the document. A quote turns checking from a research task into a ten-second glance. An answer that can't point to a passage should be treated as unsupported.

3. Make 'I don't know' an approved answer

Write the refusal into the design. When the retrieved passages don't address the question, the assistant says so and offers a path to a person. Many made-up answers come from systems that were never given permission to say nothing.

4. Set a confidence threshold

Retrieval systems score how closely each passage matches the question. Below a threshold you choose, the assistant doesn't answer; it sends the question to staff along with the passages it found. Start strict and loosen only when testing shows it's safe.

Keep in mind what the score measures: how similar a passage is to the question, not whether it's true or current. An outdated policy can match perfectly, which is why the threshold works alongside quotes and review, not instead of them.

5. Test with real questions before launch

Use questions people actually asked, taken from your inbox, phone notes and staff, not questions written to make a demo shine. Include questions the assistant should decline.

6. Review misses every week

Log every question, answer, source and refusal. Once a week, someone who knows the subject reads the refusals, the low-confidence answers and a random sample of the rest, and fixes problems in the source document. Five patterns are worth looking for:

  • Refusals on common questions: a sign a document is missing or poorly titled.
  • Answers quoting an old version: archive the outdated file so search can't find it.
  • Two sources that disagree: the owner decides which is right, and the other goes.
  • Answers that drift beyond the quote: tighten the instruction to stay within the passage.
  • Questions that should never be automated: add them to the always-route-to-a-person list.

Flowchart

How a grounded assistant decides whether to answer

  1. StartParalegal asks when the response to discovery in the Ortiz matter is due
  2. StepAssistant searches the Ortiz matter file and the firm's docketing notes
  3. DecisionIs the best match score above the threshold?
    No
    A personIt shows what it found and sends the question to the docketing clerk
    Yes
  4. DecisionDoes the passage state the deadline for this matter?
    No
    A personIt says no docketed date was found and flags the responsible attorney
    Yes
  5. StepIt answers, quoting the docket entry with a link
  6. DoneParalegal checks the quote; the attorney still owns the deadline
Every answer has to pass two gates, a relevant source and a confidence score, before anyone relies on it.

Before and after

The same question, two kinds of assistant

General-purpose chatbot

  1. 01Asked when a discovery response is due
  2. 02Answers from general training, with no access to the file
  3. 03Gives a confident, generic deadline
  4. 04Offers no source, so the answer reads as authoritative

ResultPlausible, unverifiable, possibly wrong

Grounded assistant

  1. 01Searches the matter file and docket notes
  2. 02Finds the docket entry, or says it can't
  3. 03Quotes the entry and links to it
  4. 04Sends missing or low-confidence answers to a person

ResultRight with a source, or an honest 'not found'

An open chatbot answers from memory; a grounded assistant answers from your file or says it can't.

A test plan you can run this week

Before an assistant answers anyone, run it against a fixed set of real questions and score the results. The same set becomes your regression test every time documents or settings change.

  1. 01Collect 50 real questions from the last two months: roughly 35 your documents should answer and 15 they shouldn't.
  2. 02For each, write the expected answer and its exact source, or 'should decline.'
  3. 03Run all 50 and score every answer with the table below.
  4. 04Fix each wrong answer at the source: a missing document, an outdated version or an unclear passage.
  5. 05Re-run the full set after every fix. Launch only when no high-stakes question gets a confident wrong answer.
How to score each test answer
ResultWhat it meansWhat to do
Correct, with sourceMatches the expected answer and quotes the right passageNothing
Correct, no sourceRight answer with nothing to verify it againstCount it as a failure; fix retrieval or the document
Declined correctlyYour sources don't cover it, and it said soNothing, or write a document if the question is common
Declined wronglyThe answer was in your documents but it missed itImprove the document's title, structure or wording
WrongA confident answer that doesn't match the sourceStop and trace it; this is the failure that matters

A medical practice can run the same plan for staff-facing questions such as prior authorization rules, scheduling protocols and billing policies. Clinical content stays behind clinician review: a drafted visit summary or patient reply stays a draft until a clinician has signed off.

What stays with a person

None of these controls remove the need for judgment. Decide in writing which answers the assistant may give on its own and which it may only draft. In a law firm, deadlines, citations and anything resembling advice are drafts for a lawyer. In a medical practice, anything clinical goes to a clinician.

If your sources disagree with each other, fix that first; our guide to building a knowledge base for an assistant explains how.

Our post on keeping people in the loop covers where to draw the line, and our custom AI assistants are built around the six controls above.

Sources

Common questions

Quick answers.

Why does AI give wrong answers so confidently?

Language models generate text by predicting likely words, so a wrong answer comes out just as fluent as a right one. Without a source to answer from, the model fills the gap with something plausible. Grounding answers in your own documents, with quotes, is the most effective fix.

Can AI hallucinations be eliminated completely?

No. You can make them rare and easy to catch by limiting the assistant to your sources, requiring quotes, letting it decline and reviewing misses weekly. Independent testing found that even retrieval-based legal research tools still made errors, so human review stays part of the process.

How do I test whether an AI assistant is reliable?

Build a set of about 50 real questions with known answers and sources, including some it should decline. Score each answer as correct with a source, correct without one, declined correctly, declined wrongly or wrong. Re-run the set after every change.

Can I ask the AI to double-check its own answer?

Not as your only check. In Mata v. Avianca, a lawyer asked ChatGPT whether a case it had supplied was real, and it said yes. Verification has to come from the source document or an authoritative database, not from the model.

Put it to work

Custom AI chatbots & assistants

Custom AI chatbots and assistants grounded in your policies, documents and systems: website chat for customers and an internal assistant your team trusts.

See how it works

Most relevant for law firms, doctors.

Keep reading

Ready when the work is.

Show us what is slowing your business down.

Book a call

Not ready to talk?

Written brief

Skip the call.
Send a brief.

A few sentences is plenty. We read every brief and reply by email within one business day.

Try it free. If you love it, keep it. If it's not for you, we take it back, no questions asked.

We only use this to reply to you. Privacy policy